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Executive Summary 


The purpose of this evaluation of Scholastic’s System 44 conducted by RMC Research was to 
expand the existing research on students with learning disabilities by conducting a randomized 
study of struggling readers with approximately half of the sample comprised of students with 
learning disabilities. Specifically, this evaluation examined the impact of System 44 on the 
reading outcomes of struggling readers and on a subsample of students with learning 
disabilities in Grades 4-8. 


Scholastic’s System 44 is a foundational reading program intended for older struggling readers 
who have not mastered basic phonics and decoding skills. Combining researched-based 
phonics instruction with adaptive technology, System 44 is designed to improve students’ word 
reading accuracy, fluency, and comprehension. The System 44 program delivers research- 
based instruction through an adaptive computer component; teacher-led small-group instruction; 
and individual student practice involving high-interest, leveled materials. Thus students who 
have not responded to classroom reading instruction may benefit from the more intensive and 
specific decoding instruction provided through System 44. 


The evaluation of the implementation and impact of System 44, which involved 12 elementary 
schools and 4 middle and K-8 schools in a district in Michigan, sought to answer 7 research 
questions: 


1. What contextual factors are involved in the implementation of System 44 (i.e., factors 
that promote or hinder successful implementation of the program)? 

2. What do teachers report to be the most valuable features of System 44? Does this vary 
for different subgroups of students? 

3. What are the effects of System 44 on student outcomes? Specifically, how do changes 
in word-level accuracy, fluency, and reading comprehension achieved by System 44 
students compare to changes achieved by the students in the services-as-usual control 
group? 

4. What are the effects of System 44 on outcomes of students with learning disabilities? 
Specifically, how do changes in outcomes achieved by System 44 students with learning 
disabilities compare to changes achieved by students with learning disabilities in the 
services-as-usual control group? 

5. How does System 44 differentially affect other subgroups of students? Specifically, how 
do changes in word-level accuracy, fluency, and reading comprehension achieved by 
specific subgroups of System 44 students (based on gender, ethnicity, economic status, 
English language proficiency, school level, and initial reading ability), compare to 
changes achieved by equivalent students who did not use the program? 

6. What is the association between System 44 effects and program implementation—are 
changes in System 44 participants’ word-level accuracy, fluency, and reading 
comprehension skills associated with variation in program implementation (including 
total time on software, total topics mastered, total series completed, time in small 
groups, resources utilized by students)? 

7. Does progress in the System 44 software vary by students’ special education 
classification and initial reading ability? 


The evaluators selected the target sample based on student performance on the fall 2011 
Michigan Educational Assessment Program (MEAP) and spring 2011 AlMSweb assessment. 
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The Scholastic Reading Inventory (SRI) was used to screen students for System 44 eligibility. 
The district administered the SRI to all students in the target sample. Those students who 
scored below 600 Lexiles' on the SRI were administered the Scholastic Phonics Inventory 
(SPI). All students who scored in the Beginning or Developing reader categories on the SPI 
were randomly assigned (stratified by school and grade level) to either the System 44 treatment 
group or the control group. A total of 368 students met the eligibility criteria for System 44 and 
were randomly assigned to the treatment group (n = 187) or the control group (n = 181). The 
groups were equivalent in terms of sex, eligibility for free or reduced-price meals, English 
proficiency, special education status, ethnicity, and baseline MEAP scores. Of the randomly 
assigned students, 155 in the treatment group and 162 in the control group received the 
allocated intervention or control group condition as planned. A total of 195 students (53% of the 
students who were randomly assigned) had a designated learning disability. 


RMC Research hired and trained 4 local testers to individually administer a battery of 
standardized reading tests to all treatment and control group students. The testers administered 
the tests in October 2011 to establish baseline scores and again in May 2012 to attain follow-up 
scores. The tests included the following: 


« Test of Silent Reading Efficiency and Comprehension (TOSREC). 
«Comprehensive Test of Phonological Processing (CTOPP) Elision subtest. 

= Test of Word Reading Efficiency (TOWRE) Sight Word Efficiency subtest. 
«Test of Word Reading Efficiency (TOWRE) Phonetic Decoding Efficiency subtest. 


Other tests administered to the participating students included the SRI, a test of reading 
comprehension skills; the SPI, a test of accuracy and fluency in letter recognition, sight word 
recognition, and non-word decoding; and the English-Language Arts subtest of the MEAP. The 
evaluation team also collected implementation data through fall and spring teacher surveys; 
spring teacher and district staff interviews; fall, winter, and spring classroom observations; and a 
professional development observation. 


Program Implementation Findings 


The evaluation team assessed the fidelity of System 44 implementation in 2 ways: classroom 
observations and teacher self-report in surveys and interviews. 


«= Classroom Setup (accessibility and functioning of computers, headphones, 
microphones, CD players, and print materials): In fall 2011, 36% of the 20 classrooms 
observed received the highest rating; this percentage increased to 67% of the 
classrooms in winter 2012 and 78% in spring 2012. 

« Minutes of Instruction: Across the 3 observation points, from 72 to 82% of the 20 
classrooms provided at least 55 minutes of instruction daily. 

« Inclusion of Program Components: In most cases, half of the class used the computer 
software while the other half received small-group instruction or worked independently, 
and midway through the class period the groups changed places. During teacher-led 
small-group activities, observers most frequently noted the use of the System 44 flip 
chart, the Decodable Digest, and System 44 paperback books. While students worked 
independently, observers most frequently noted the use of System 44 paperback books, 
System 44 audiobooks, and Scholastic Achievement Manager (SAM) worksheets. Level 
of student engagement showed no consistent pattern across the 3 observation periods. 

« Instructional Management and Delivery: Overall, the observers rated instructional 
management highest in fall 2011 and rated instructional delivery highest in winter 2012. 


‘For more information about the Lexile Framework for Reading see http://www.lexile.com/ 
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The aspects of instructional management that received the lowest ratings were 
“differentiate support based on students’ needs” and “use flexible groupings for students 
based on instructional needs.” The aspects of instructional delivery that received the 
lowest ratings were “appear well prepared for the lesson” and “execute the lesson well.” 

= Teacher Understanding of System 44 Program Components: At the conclusion of 
the System 44 training in September 2011 and again in spring 2012, the participating 
teachers were asked to respond to 10 questions designed to test their understanding of 
the System 44 program components. On average, the teachers answered 65% of the 
items correctly in fall 2011 and 76% of the items correctly in spring 2012. This 
improvement was statistically significant (p < .05). 

«= Teacher Ratings of Instructional Practices: The System 44 teachers were more likely 
than the control group teachers to work one-to-one with students; work with small groups 
of students; group students based on skill levels; group students based on need for 
additional instruction in specific skills; use reading software, teach specific strategies for 
decoding unfamiliar words; teach decoding/phonics skills while reading stories; have 
students practice reading high-frequency words for automaticity; and have students use 
knowledge of root words, prefixes, and suffixes to decode new words. 

« Whole-Group Instruction: Most of the 20 teachers (70%) used whole-group instruction 
every day; the remaining 30% said they did so 2 or 3 times a week. 

« Factors That Facilitated Implementation: The 18 System 44 teachers and 2 district 
staff who were interviewed for the evaluation identified 2 primary factors that facilitated 
program implementation: professional development and coaching from Scholastic staff 
and the System 44 software and printed materials. 

= Factors That Were Barriers to Implementation: The teachers and district staff 
identified 3 primary barriers to implementation: problems with technology, problems with 
student behavior, and the difficulties inherent in learning how to teach a new program. 


Student Outcomes 


RMC Research used multilevel models to estimate the impact of System 44 on spring 2012 
student outcome scores while controlling for fall 2011 scores, special education status, ethnicity, 
English language learner status, and school level. The treatment group students performed 
significantly better than the control group students on 2 of the 4 individual tests: CTOPP Elision 
and TOWRE Sight Word Efficiency, with effect sizes of .27 and .16, respectively. 


Multilevel modeling on SRI and SPI outcomes also showed positive results for the overall 
sample. The impact was significantly greater for the treatment group students than the control 
group students on the SRI (effect size = .32). In addition, the treatment group students 
performed significantly better than the control group students on SPI Nonsense Word Accuracy 
and SPI Overall Accuracy (effect sizes .32 and .16, respectively). Differences between the 2 
groups were not significant for other SPI subtests. 


The hierarchical linear models conducted for each of the 4 individual reading test outcomes and 
the SRI and SPI outcomes consistently revealed main effects for learning disability. Specifically, 
learning disabled students scored significantly lower than non-learning disabled students on all 
tests except for the SPI Letter Name Accuracy test. In addition, multilevel modeling revealed 
significant impacts on the same 2 tests (CTOPP Elision and TOWRE Sight Word Efficiency) that 
yielded significant impacts in the overall sample; however, the effect sizes for the learning 
disabled sample were larger than for the overall sample on both tests (.36 and .24 respectively). 
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Although the effects on learning disabled students were slightly stronger, the impact of 
System 44 was statistically equivalent for learning disabled and non-special education students. 


RMC Research also examined the moderating effects of demographic characteristics. Ethnicity 
did not moderate treatment effects—that is, the effects of System 44 were similar for African 
American and non-African American students. The results did, however, show that school level 
moderated the effects of System 44 on several tests: TOSREC, SRI, SPI Sight Word Accuracy, 
SPI Decoding Accuracy, and SPI Overall Accuracy. Specifically, student gains on these tests 
were greater at the middle school level than at the elementary school level. 


The results of the implementation analyses showed some variation in implementation across 
schools. The evaluation team conducted nonexperimental analyses to address the question: To 
what extent is program implementation associated with treatment impact? The data collected 
from the teacher surveys and classroom observations were used to develop 4 teacher-level 
variables: total number of times the teacher used System 44 materials during small-group 
instruction, average number of students using System 44 paperback books or the Decodable 
Digest during independent work time, instructional management, and instructional delivery. 
However, none of these variables were associated with student outcome measures. 


At the point of program exit, System 44 students should have completed all 160 topics covered 
in the 25 series that compose the software component of the program. The analysis conducted 
by the evaluation team revealed that 21 of the 155 System 44 students (14%) completed all 160 
topics (average number of topics completed: 77). On average, students with learning disabilities 
completed 62 topics compared to non-special education students who completed 95 topics. 
Further, students who completed fewer than 40 topics spent significantly fewer hours on the 
software than students who completed between 40 and 100 topics or between 100 and 160 
topics. 


A student's initial decoding status on the SPI determines the starting series in System 44 and is 
an indicator of students’ initial reading ability. Seventy-one students were classified as 
beginning decoders and 82 students were classified as developing decoders at baseline. The 
analysis of variance (ANOVA) results showed significant differential gains between beginning 
and developing decoders on the following tests: SPI Sight Word Fluency (p < .05), and SPI 
Overall Fluency (p < .05). Specifically, beginning decoders had significantly greater gains on 
these 2 fluency tests than developing decoders. 


In summary, the evaluation of System 44 revealed significant impacts on several tests for both 
the overall sample and the learning disabled sample. Additional findings revealed that impacts 
were stronger on several tests for middle school students than for elementary school students, 
particularly on SPI Nonsense Word Accuracy, TOSREC, and SRI. Although significant impacts 
were attained by the end of Year 1, the majority of students in the study did not complete the 
System 44 program. Data collected through teacher surveys, classroom visits, and interviews 
provided information on teachers’ implementation of System 44 in the classroom, and software 
usage data were used to examine differences in students with varying program exit and topic 
completion patterns. 
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Evaluation Goals and Objectives 


Effective reading instructional practices for beginning and struggling readers have been the 
topic of a significant amount of research over the past decade. The authors of a 1998 report 
published by the National Research Council (Snow, Burns, & Griffin, 1998) concluded that 
systematic and explicit phonics instruction is an effective means of teaching children to read at 
the word level. In 2000 the National Reading Panel conducted a meta-analysis of quantitative 
studies of phonics instruction and determined that systematic phonics instruction helped 
students learn to read better than all forms of control group instruction (National Institute of 
Child Health and Human Development, 2000). The panel concluded that systematic and explicit 
phonics instruction should be incorporated into literacy programs for beginning readers and into 
programs for preventing and remediating reading difficulties in struggling readers. Vaughn, 
Wanzek, Woodruff, and Linan-Thompson (2007) suggested that to improve outcomes for 
students at risk of reading failure, schools need to incorporate best practices in critical areas: 
effective reading instruction for all students; early identification of students at risk for reading 
problems; effective interventions for students at risk; and professional development for teachers. 
Since 2000 numerous instructional programs that incorporate scientific reading research have 
been developed. In general, the effectiveness of these programs has been difficult to ascertain. 
Many have not undergone evaluations, and many of the evaluations that have been conducted 
employed research designs that lacked rigor. 


Because students with learning disabilities may be at particular risk of reading failure, the need 
for rigorous evaluations measuring the effectiveness of reading interventions on students with 
learning disabilities is even greater. Two recent syntheses of experimental research reported 
promising results regarding intervention effectiveness. A synthesis of reading comprehension 
interventions for middle school students with learning disabilities examined studies with 
treatment and comparison designs: researcher-developed comprehension measures yielded 
large effect sizes, whereas a few studies that reported standardized measures of reading 
comprehension indicated medium effect sizes (Solis et al., 2012). Another meta-analysis 
focused on reading interventions for upper elementary and middle school students with reading 
disabilities using norm-referenced reading measures. Aggregated across 10 studies conducted 
between 1994 and 2009, moderate effect sizes were observed for norm-referenced measures of 
word identification, decoding, and comprehension, and low effect sizes were observed for 
fluency. Outcomes did not significantly vary based on type of reading instruction or sample 
characteristics (Flynn, Zheng, & Swanson, 2012). 


Two recent studies compared combinations of reading interventions or multiple-component 
interventions with so-called “ordinary special education” or phonological control programs for 
students with reading disabilities; in both studies, combination interventions showed significant 
improvements relative to control programs (Gustafson, Faith, Svensson, Tjus, & Heimann, 
2011; Morris et al., 2012). Equivalent gains in the Morris et al. (2012) study were observed for 
different racial, socio-economic status, and IQ groups. A multiple-baseline design with Grade 2 
students targeting word decoding performance demonstrated substantial gains in decoding and 
reading accuracy, which were maintained during follow-up observations (Cohen & Brady, 2011). 


Although recent studies have yielded promising results, the sample sizes in many of the cited 
studies were small; the average sample size across the 10 studies included in the meta-analysis 
conducted by Flynn et al. (2012) was 51, and the sample size of the Gustafson et al. (2011) 
study was 50. In addition, not all research has been uniformly favorable. For example, a 
randomized experimental design tested the effects of a fluency intervention program on word 
identification and reading comprehension of middle school students with severe reading delays. 
Students in the experimental group made more progress on reading fluency than control group 
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students, yet had no gains in reading comprehension (Spencer & Manis, 2010). The literature 
on reading interventions for students with reading disabilities suggests that the understanding of 
intervention effectiveness continues to evolve and indicates a continued need for rigorous large- 
scale studies of reading interventions on this population of struggling readers. 


The purpose of this evaluation of Scholastic’s System 44 conducted by RMC Research was to 
expand the existing research on students with learning disabilities by conducting a randomized 
study of struggling readers with approximately half of the sample comprised of students with 
learning disabilities. Specifically, this evaluation examined the impact of System 44 on reading 
outcomes of struggling readers and on a subsample of students with learning disabilities. 


Intervention Model 


Scholastic’s System 44 is a recently developed foundational reading program intended for older 
struggling readers who have not mastered basic phonics and decoding skills. Combining 
research-based phonics instruction with adaptive technology, System 44 is designed to improve 
students’ word reading accuracy, fluency, and comprehension. The System 44 program delivers 
research-based instruction through an adaptive computer component; teacher-led small-group 
instruction; and individual student practice involving high-interest, leveled materials. Thus 
students who have not responded to classroom reading instruction may benefit from the more 
intensive and specific decoding instruction provided through System 44. 


Students in the System 44 classroom are expected to receive approximately 20 to 25 minutes of 
computer-delivered instruction, complete 25 to 30 minutes of small-group and individual work, 
and receive 5 to 10 minutes of whole-class instruction each day. The program includes 25 
series, each of which covers 5 to 8 topics, each of which comprises 4 strands: The Code 
(decoding instruction), Sight Words (high frequency words), Word Strategies (syllable types and 
word parts), and Success (reading connected text). Beginning with Series 4, students complete 
a preliminary Fast Track Assessment to determine whether they have already mastered the 
skills covered in the series and can skip to the next series. Each software lesson in a series has 
a set of corresponding practice activities in (a) the 44Book, (b) the Decodable Digest, and 

(c) the 36-title System 44 paperback book and audiobook library. Supplementary instructional 
materials include practice worksheets from the online Scholastic Achievement Manager (SAM), 
letter tiles, posters, flip chart lessons, and the DVD Sound and Articulation. Students track their 
progress on these materials using the Self-Monitoring Chart. Approximately twice a week the 
teacher leads a 5- to 10-minute whole-class instructional activity. The teacher can use SAM to 
generate reports to track individual student progress on each skill and identify students in need 
of small-group instruction. 


Logic Model 


Exhibit 1 portrays a logic model that summarizes the System 44 program inputs, classroom 
practices, and expected outcomes for participating teachers and students. 


Overview of the Evaluation 


Using a randomized design, this evaluation assessed the effectiveness of System 44 in terms of 
improving the foundational reading skills of struggling readers in Grades 4—8 in an urban school 
district in Michigan that served approximately 9,000 students in 21 schools: 15 elementary 
schools, 2 middle schools, and 2 high schools, and 2 K-8 schools. The evaluation of the 
implementation and impact of System 44, which involved 12 of the elementary schools and all 4 
middle and K-8 schools in the district, sought to answer 7 research questions: 


1. What contextual factors are involved in the implementation of System 44 (i.e., factors 
that promote or hinder successful implementation of the program)? 
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2. What do teachers report to be the most valuable features of System 44? Does this vary 
for different subgroups of students? 

3. What are the effects of System 44 on student outcomes? Specifically, how do changes 
in word-level accuracy, fluency, and reading comprehension achieved by System 44 
students compare to changes achieved by the students in the services-as-usual control 
group? 

4. What are the effects of System 44 on outcomes of students with learning disabilities? 
Specifically, how do changes in outcomes achieved by System 44 students with learning 
disabilities compare to changes achieved by students with learning disabilities in the 
services-as-usual control group? 

5. How does System 44 differentially affect other subgroups of students? Specifically, how 
do changes in word-level accuracy, fluency, and reading comprehension achieved by 
specific subgroups of System 44 students (based on gender, ethnicity, economic status, 
English language proficiency, school level, and initial reading ability), compare to 
changes achieved by equivalent students who did not use the program? 

6. What is the association between System 44 effects and program implementation—are 
changes in System 44 participants’ word-level accuracy, fluency, and reading 
comprehension skills associated with variation in program implementation (including 
total time on software, total topics mastered, total series completed, time in small 
groups, resources utilized by students)? 

7. Does progress in the System 44 software vary by students’ special education 
classification and initial reading ability? 
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Exhibit 1 
Logic Model for System 44 


Program Classroom Teacher Short-Term Long-Term 
Inputs/Activities Practices and Outcomes Student Patil 
ay Outcomes utcomes 
Professional - biel Fidelity of System 44 
development for teachers implementation: Total System 44 Improved reading 
(beginning of year). Daily use of * Classroom topics completed. fluency and 
In-school coaching by Scholastic setup. Improved decoding Comprenension: 
Scholastic (3 sessions System 44 software = Minutes of accuracy, sight word Improved 
per teacher). with fidelity. instruction. accuracy, and performance on 
i state reading test. 
District meetings for Whole-class = Use of program ee g 
System 44 teachers to Haat components. 
share experiences. = 
; P ; beginning of class. Student 
Curriculum materials: Se lara grouping. 
« System 44 software. Caan, Satisfaction with 


professional 
development and 
support. 


« Decodable Digest. 
= 44Book. 


Differentiated 
instruction. 


« System 44 paperback Opportunities for 
! Perceived 
book and audiobook students to practice effectiveness of 
library. skills independently. program. 


* Other (practice 
worksheets, letter 
tiles, posters, flip 

chart, Sound and 

Articulation DVD). 
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Evaluation Methods 


Scholastic implemented System 44 in 12 elementary, 2 middle, and 2 K-8 schools in an urban 
district in Michigan during the 2011-2012 school year. The evaluation used a randomized trial 
design whereby RMC Research randomly assigned eligible students to a treatment (System 44) 
group or a control (“services-as-usual”) group. 


Sampling Plan 


The evaluators selected the target sample based on student performance on the fall 2011 
Michigan Educational Assessment Program (MEAP) and spring 2011 AlMSweb assessment. 
Students who performed below the 50" percentile on the fall 2011 MEAP or below the 25" 
percentile on the spring 2011 AlMSweb assessment were subsequently tested for System 44 
eligibility. The evaluators purposefully sampled a large proportion of students with learning 
disabilities. Thus all students in Grades 4-8 in the participating schools with a classified learning 
disability who met the MEAP or AlMSweb eligibility criteria were tested for System 44 eligibility. 
Students with learning disabilities comprised approximately half of the randomized sample. The 
remainder of the sample was comprised of non-special education students in Grades 4-8. To 
identify the non-special education sample, non-special education students who met the MEAP 
or AlMSweb eligibility criteria were stratified into deciles by their MEAP or AlMSweb 
assessment score. The evaluators randomly selected 20 students from within each decile to test 
for System 44 eligibility. 


After the target sample was established, a 2-step process was used to establish student 
eligibility for System 44. The Scholastic Reading Inventory (SRI) was used to screen students 
for System 44 eligibility. The district administered the SRI to all students in the target sample. 
Those students who scored below 600 Lexiles* on the SRI were administered the Scholastic 
Phonics Inventory (SPI), a computer-based test used to identify students in need of additional 
phonics instruction. All students who scored in the Beginning or Developing reader categories 
on the SPI were randomly assigned (stratified by school and grade level) to either the 
System 44 treatment group or the control group. 


Participants 


A total of 368 students met the eligibility criteria for System 44 and were randomly assigned to 
the treatment group (n = 187) or the control group (n = 181). A total of 195 students (53% of the 
students who were randomly assigned) had a designated learning disability. Exhibit 2 
summarizes the characteristics of all treatment and control group students in the randomized 
sample and those students who were included in the analytic sample. The evaluation team 
conducted equivalence tests on key factors to determine whether differences between the 
treatment and control groups existed at baseline. Appendix A provides baseline equivalence 
test results for the analytic sample. Overall, the treatment and control groups in both the 
randomized and analytic samples were equivalent in terms of sex, eligibility for free or 
reduced-price meals, English language proficiency, special education status, ethnicity, and 
baseline MEAP scores. 


*For more information about the Lexile Framework for Reading see http://www.lexile.com/ 
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Exhibit 2 
Student Characteristics 


Randomized Sample Analytic Sample 
Treatment Control Treatment Control 

Characteristic (n = 187) (n = 181) (n = 155) (n = 162) 
Grade Level 

4 37 (20%) 0 (17%) 1 (20%) 8 (17%) 

5 58 (31%) 5 (36%) 4 (35%) 9 (36%) 

6 45 (24%) 40 (22%) 33 (31%) 5 (22%) 

7 23 (12%) 9 (10%) 9 (12%) 7 (10%) 

8 24 (13%) 7 (15%) 8 (12%) 3 (14%) 
Sex 

Male 114 (61%) 108 (60%) 94 (61%) 97 (60%) 

Female 73 (39%) 73 (40%) 61 (39%) 65 (40%) 
Free or Reduced-Price Meals 

None 9 (5%) 9 (5%) 7 (4%) 6 (4%) 

Free or reduced price 178 (95%) 172 (95%) 148 (96%) 156 (96%) 
English Learner Status 

Non English learner 180 (96%) 172 (95%) 149 (96%) 153 (94%) 

English learner 7 (4%) 9 (5%) 6 (4%) 9 (6%) 
Special Education Status 

None 86 (46%) 92 (51%) 71 (46%) 81 (50%) 

Specific learning disability 97 (52%) 88 (49%) 84 (54%) 81 (50%) 

Other 4 (2%) 1 (1%) 0 (0%) 0 (0%) 
Primary Ethnicity 

Caucasian 24 (13%) 9 (10%) 16 (10%) 17 (10%) 

African American 146 (78%) 141 (78%) 124 (80%) 124 (77%) 


Hispanic 17 (9%) 


1 (12%) 15 (10%) 21 (13%) 
Note. Randomized sample n = 368; Analytic sample n = 317. 


Of the 368 randomly assigned students, 317 (155 treatment, 162 control) received the allocated 
intervention or control group condition as planned. Of the 32 treatment group students and 19 
control group students who did not receive the allocated intervention or control group conditions 
as planned, 8 treatment and 1 control group student were misclassified as eligible for the 
intervention and study at the time of random assignment and should be excluded from attrition 
analyses. Reasons for exclusion include: skills deemed too high for the intervention after 
random assignment occurred (3 treatment), inappropriate assignment of a special education 
designation other than learning disability (4 treatment, 1 control), and non-advancement to 
Grade 4 (1 treatment). A total of 34 students (18 treatment, 16 control) were eligible for the 
intervention and study but did not receive the allocated intervention or control group condition 
for the following reasons: 22 (11 treatment, 11 control) transferred to a non-System 44 school, 9 
were withdrawn by their parents (6 treatment, 3 control), 2 encountered irreconcilable 


6 Evaluation of System 44 


scheduling issues (1 treatment, 1 control), and 1 was homebound for medical reasons (1 
control). Additionally, 8 students who received the allocated treatment or control condition were 
excluded from the analytic sample: 1 treatment group student was not administered pretests 
because the test administrator was unable to locate the student, and 7 students (5 treatment, 2 
control) were not administered posttests because they were on long-term suspension through 
the spring 2011 testing period. A total of 317 of the 368 randomly assigned students (86%) were 
retained in the final analytic sample for all individual test outcomes. The final SRI analytic 
sample was 287 and the final SPI analytic sample was 279.° 


To assess whether selective study attrition occurred in the analytic sample, the evaluation team 
conducted equivalence tests on baseline demographic characteristics and MEAP reading 
scores. No differences were evident at baseline in terms of sex, eligibility for free or 
reduced-price meals, special education classification, and English language learner 
classification between students who were retained in the analytic sample and students who 
were not. However, students retained in the final analytic sample were more likely than students 
who were not retained to be Hispanic. 


Data Collection 


Data collection activities for the System 44 evaluation included student reading tests, teacher 
surveys, System 44 classroom observations, a professional development observation, and staff 
interviews. Survey, observation, and interview materials are included in Appendix B. 


Student Reading Tests 


RMC Research hired and trained 4 local testers to administer a battery of standardized reading 
tests to all treatment and control group students. The testers administered the tests to each 
student separately over a 3-week period in October 2011 to establish baseline scores and again 
in May 2012 to attain follow-up scores. The tests included the following: 


« Test of Silent Reading Efficiency and Comprehension (TOSREC)—A 60-item 
comprehension test that assesses a student’s ability to silently read a sentence and 
identify whether the sentence is true or false. The test is grade level specific. Form A 
was administered in fall 2011, and Form C was administered in spring 2011. 

= Comprehensive Test of Phonological Processing (CTOPP) Elision subtest—A 
34-item test that assesses the extent to which a student can say a word and then say 
what is left of the word after being instructed to drop out designated sounds. 

« Test of Word Reading Efficiency (TOWRE) Sight Word Efficiency subtest—A 
108-item test that measures the number of printed words that a student can accurately 
read aloud in 45 seconds. Form A and Form B were administered back-to-back in fall 
2011 and spring 2011. 

« Test of Word Reading Efficiency (TOWRE) Phonetic Decoding Efficiency subtest— 
A 66-item test that assesses a student’s ability to apply grapho-phonemic knowledge by 
reading pronounceable non-words. The test measures the number of non-words that a 
student can accurately read aloud in 45 seconds. Form A and Form B were administered 
back-to-back in fall 2011 and spring 2011. 


The battery of standardized reading tests required approximately 20 to 25 minutes to administer 
to each student, which included time to build rapport. The test administrators reported that the 
testing went smoothly overall. Other tests administered to the participating students included the 


3A total of 30 students were excluded from the SRI analytic sample due to either a baseline test date that occurred 
mid-year or a follow-up test that occurred mid-year. A total of 26 students were excluded from the SPI analytic 
sample due to a follow-up test that occurred mid-year. 
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SRI, a test of reading comprehension skills; the SPI, a test of accuracy and fluency in letter 
recognition, sight word recognition, and non-word decoding; and the English-Language Arts 
subtest of the MEAP.* 


Teacher Survey 


A survey administered to System 44 teachers in fall 2011 and spring 2012 gathered information 
about the teachers’ background, the types of instructional strategies and activities they 
employed to teach foundational reading skills, their opinions about the effectiveness of 

System 44, their opinions about System 44 professional development, and their knowledge of 
the System 44 program. For comparative purposes, a sample of control group teachers 
completed a modified version of the survey in spring 2012. Teachers who taught both 

System 44 and control group students answered additional questions about instructional 
materials and types of assistance available to control group students. 


Classroom Observations 


RMC Research staff observed every System 44 classroom 3 times during the 2011-2012 
school year (November, February, and May) to gather data on System 44 implementation. The 
classroom visit protocol captured the essential features of System 44 and other features of 
effective instruction such as maintaining a positive learning environment, monitoring students, 
and providing differentiated support based on student needs. 


Professional Development Observations 


RMC Research staff also observed the professional development provided to System 44 
teachers at the beginning of the school year to better understand the expected implementation 
of the program and any training issues that might have interfered with the fidelity of program 
implementation. 


Staff Interviews 


The end-of-year interviews with teachers, principals, and district staff assessed the contextual 
environment in which System 44 was implemented, the factors that facilitated or hindered 
implementation, the fidelity of System 44 implementation, and staff perceptions of System 44 
materials and instructional activities. 


Analysis Plan 


RMC Research utilized a mixed-method approach to evaluate the impact of System 44. 
Program implementation was evaluated through quantitative analyses of classroom 
observations and teacher surveys; these analyses were supported by qualitative information 
gathered through interviews with district staff, school principals, and System 44 teachers. To 
measure the impact of System 44 on student test scores, RMC Research used a series of 
quantitative analyses. 


Implementation Analysis 


Classroom implementation fidelity was defined as the average of the fidelity ratings that were 
part of the classroom observations conducted by the evaluation team in fall 2011, winter 2011, 
and spring 2012. 


“Follow-up MEAP scores, which will be administered in fall 2012, were not available at the time of this report. Thus 
MEAP outcomes are not presented in this report. 
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Implementation fidelity included the following components: 


» Classroom setup. 

« Minutes of instruction on software and in small groups. 
» Inclusion of all program components. 

» Instructional management and delivery. 


Teacher survey data were used to examine teachers’: 


» Background characteristics. 

« Perceptions of System 44 professional development. 

» Reading instructional practices prior to System 44 compared to reading instructional 
practices using System 44. 

« Reading instructional practices of System 44 teachers compared to reading instructional 
practices of control group teachers. 

«Perceptions of the effectiveness of reading instructional practices prior to and using 
System 44. 

= Frequency of use of System 44 components. 

« Understanding of System 44 program concepts and components. 


Teacher interview data from spring 2012 provided detailed information concerning the teachers 
perceptions of the strengths and challenges of using System 44, how they made decisions 
concerning student grouping, and how they used various SAM reports. Principal interview data 
from spring 2012 provided information about the strengths and challenges of implementing 
System 44 and other contextual issues. District staff interview data from spring 2012 provided 
additional information about the professional development, the factors that facilitated and 
hindered implementation of System 44, perceptions of program effectiveness, and 
recommendations for future implementation. 


Impact Analysis 


This evaluation used an intent-to-treat statistical model—that is, a framework in which 
participants are analyzed within their initial random assignment group regardless of whether 
they actually received treatment. Because students were clustered within schools, a multilevel 
model was used to estimate the impact of the intervention on spring 2012 outcome scores while 
controlling for baseline score, ethnicity, special education status, English language proficiency, 
and school level. The model was run separately for each outcome measure. The district 
provided individual student demographic data for grade level, sex, free or reduced-price meal 
eligibility, ethnicity, special education status, and English language proficiency. The following 
demographic covariates were included in the model: ethnicity (Caucasian), special education 
status, and sex. In addition, school level was added as a covariate. No data were missing for 
any of the demographic covariates. To assess the impact of System 44 on student outcomes, 
the evaluation team used hierarchical linear modeling, controlling for baseline scores and 
student demographic characteristics at Level 1 and school at Level 2. The 2-level model for 
estimating the impact of System 44 on change in outcomes follows: 
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Level 1 Model: 
Yi = Boj + B4)(Trtj) + B2i(Pretest;) + B3(LDj) + B,4j(Ethnicity;) + B5(ELLi) + Bg(Leveli) + &j 


Level 2 Model: 
Bo; = Yoo + Ho; 
Bij = Y10 
Boj = Y20 
Bg; = 30 
Ba = Yao 
Bsj = 50 
Be = Yeo 

where: 


the random error term for student /in school / 


lj = the random intercept term for school / 
and: 

Yj = the posttest score for student /in school / 

Trtj = the treatment indicator for student /in school / 
(0 = control, 1 = treatment) 

Pretest; = the posttest score for student /in school / 

LD; = the special education indicator for student /in school / 
(0 = no special education classification, 1 = learning disabled 
classification) 

Ethnicity; = the ethnicity indicator for student /in school / 
(0 = non-African American, 1 = African American) 

ELL; = the English language learner indicator for student / in school / 
(0 = non-English language learner, 1 = English language learner) 

Leveli = the school level indicator for student /in school / 


(0 = middle, 1 = elementary) 


A similar 2-level model was used to assess the impact of System 44 on change in outcomes in 
the learning disabled sample; the model was the same as that specified above except for the 
removal of the special education indicator. 


Evaluation of System 44 


Program Implementation Findings 


To provide context for interpreting the impact of the System 44 intervention, the evaluation team 
assessed its implementation from several perspectives. Scholastic staff expected 
implementation to improve as teachers gained experience using System 44. Although many 
factors contributed to the fidelity of implementation, barriers to implementation also emerged. 
This section summarizes the findings pertaining to program implementation. 


Contextual Factors in System 44 |mplementation 


Data regarding contextual factors involved in the implementation of System 44 were obtained 
from the teacher surveys; the interviews of teachers and district staff; and the observations of 
System 44 classrooms. These data include descriptions of the System 44 classrooms and the 
counterfactual, teacher background characteristics, professional development activities, and the 
observed fidelity of System 44 implementation. 


System 44 Classrooms 


Scholastic staff expected teachers using the System 44 model in a 60-minute class period to 
allocate their time such that students spent approximately 20 minutes on the System 44 
software; approximately 20 minutes reading System 44 library books, completing book logs, and 
taking Scholastic Reading Counts quizzes; and approximately 20 minutes in whole-class or 
small-group instruction involving SMART lessons, the 44Book, or the Decodable Digest or 
conferencing individually with the teacher. 


The Counterfactual 


The counterfactual varied across schools because students in the control group did not receive 
a uniform alternative to the System 44 intervention. Most of the elementary schools used a 
pull-out model, but students were pulled out of a variety of classroom activities including 
language arts, physical education, art, and resource room time (for some special education 
students). One middle school used System 44 as a replacement reading class and the other 
middle school pulled students out of a study skills class. The control group students who 
remained in the regular classroom engaged in a variety of activities across content areas. Some 
control group students who were designated as special education spent time in the resource 
room of another teacher or in the resource room of the System 44 teacher but at a different time 
than the System 44 time block. Although some System 44 teachers also taught control group 
students, they did not use System 44 materials with them. Control group students received 
pull-out supplemental instruction from resource room (special education) or Title | teachers, 
in-class assistance from instructional aides, and after-school tutoring. The most common 
reading interventions included Houghton-Mifflin, Read Naturally, ReadAbout, SRA Decoding, 
Fast ForWord, SuccessMaker, and Reading A—Z. Control group students who participated in 
Resource Room instruction were exposed to a wide variety of instructional materials. 


Teacher Background Characteristics 


The teacher survey administered to both the treatment group and the control group solicited 
information about the respondents’ teaching experience, education, certifications, and prior 
experience with Scholastic’s System 44 and Read 180 programs. Most of the System 44 
teachers (96%) were female, and 89% of the control group teachers were female. Exhibit 3 
shows that the System 44 teachers had more teaching experience than the control group 
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teachers and were more likely to have a degree beyond a Bachelor's degree. The System 44 
teachers were also more likely to have full teacher certification (Professional Education 
Certificate). Both groups had a few teachers with experience teaching Read 180, and 1 control 
group teacher had previous experience teaching System 44. 


Exhibit 3 
Teacher Education and Experience 
Treatment Control 

Characteristic M SD Percent M SD Percent 
Teaching Experience 

Total years of teaching experience 13.7 8.3 11.3 6.8 

Years teaching in this district 11.7 7.4 8.4 6.5 

Years teaching at current school 5.2 5.8 6.2 5.7 
Education (Highest Degree) 

Bachelor’s degree 28% 38% 

Master’s degree 73% 59% 

Specialist or doctorate 0% 3% 
Certifications 

Interim Teaching Certificate 0% 3% 

Provisional Certificate 17% 34% 

Professional Education Certificate 83% 62% 
Prior Scholastic Experience 

System 44 0% 4% 

Read 180 11% 11% 


Note. n= 20 System 44 teachers; n= 29 control group teachers. 


Professional Development 


Professional development for the System 44 teachers included a 1-day training in September 
2011, follow-up training in November 2011, and monthly coaching visits from Scholastic staff 
from December 2011 through May 2012. Analyses of the professional development are based 
on the training observation results, information obtained from the Scholastic trainer at the 
conclusion of the training, and the teacher survey results. 


In September 2011 Scholastic staff provided a 1-day training for teachers who were likely to use 
the System 44 program. The training started with some confusion: more teachers than expected 
came to the training, and the teachers were prepared for the training to start earlier than the 
time the district had relayed to the Scholastic trainer and the evaluator. District staff arrived after 
the training began, clarified which teachers needed to participate, and released 6 teachers from 
the training. The district staff did not remain at the training. Among the 26 teachers left in 
attendance, most indicated that they were not familiar with System 44 prior to the training. 


The trainer provided a thorough overview of System 44 that sufficiently addressed the 
intervention components and the student learning goals and objectives and lightly covered the 
theoretical and empirical support for the intervention content. The trainer introduced the 
teachers to the breadth of the instructional materials, dividing them into 3 small groups that 
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rotated among stations, but did not have sufficient time to cover the materials in depth. The 
trainer addressed questions about specific implementation requirements for the evaluation 

(e.g., the System 44 class period needed to be 60 minutes long and students needed at least 20 
minutes on the computer daily) and took note of questions and concerns for the district to 
address with the teachers, such as how the teachers would gain access to computers for the 
program and the services that high-needs students selected for the control group would receive. 


The trainer addressed strategies for integrating the intervention into the classroom, utilizing the 
embedded tests and highlighting key instructional skills, but the teachers spent little time 
practicing the skills. Overall, the trainer provided a quality, interactive training that was tailored 
to the teachers’ needs, covered the breadth of the program materials, and addressed the 
teachers’ questions and concerns. A longer initial training would have allowed the trainer to 
cover the materials in greater depth and offer more time for teachers to practice key 
instructional strategies. 


At the conclusion of the training in September 2011 the teachers responded to several survey 
questions. System 44 teachers responded to survey questions about the professional 
development activities again in spring 2012. Exhibit 4 summarizes the System 44 teachers’ 
responses at both time points. Overall, the teachers gave the professional development above 
average ratings, with mean ratings somewhat higher in spring 2012 than in fall 2011. 


Exhibit 4 
Ratings of System 44 Professional Development 
Fall 2011 Spring 2012 
Professional Development Characteristic M SD M SD 
The teacher training prepared me to use the 3.56 0.78 3.80 1.20 
program in my classroom. 
The individual support from Scholastic during the 3.11 0.90 4.50 0.51 
year enhanced my skills in using System 44 in 
my classroom. 
The individual support from district staff during — — 3.25 1.41 
the year enhanced my skills in using System 44 
in my classroom. 
| am pleased with the amount of System 44 — — 3.55 1.10 
professional develop | received. 
| am pleased with quality of System 44 3.72 1.02 3.95 1.05 


professional development | received. 


Note. A dash indicates that the question was not asked at that time point. Ratings based on a 5-point scale that 
ranged from 1 (strongly disagree) to 5 (strongly agree). Number of respondents = 18. 


Fidelity of Implementation: Classroom Observations 


The evaluation team assessed the fidelity of System 44 implementation in 2 ways: classroom 
observations and teacher self-report on surveys and in interviews. This section summarizes the 
fall 2011, winter 2012, and spring 2012 classroom observations in terms of classroom setup, 
minutes of instruction, inclusion of program components, instructional management, and 
instructional delivery. 


Classroom Setup. Classroom setup was rated on 3 criteria: computers for at least one third of 
the class are accessible and functioning, auxiliary equipment (headsets, microphones, CD 
players) are accessible and functioning, and System 44 materials are easily accessible to 
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students and teachers. The mean rating across these items was converted to a 4-point scale. In 
fall 2011, 36% of the 20 classrooms observed received the highest rating; this percentage 
increased to 67% of the classrooms in winter 2012 and 78% in spring 2012. 


Minutes of Instruction. Average minutes of System 44 instruction were calculated for the fall 
2011, winter 2012, and spring 2012 observations and ranged from 56 to 58 minutes. From 72% 
to 82% of the 20 classrooms provided at least 55 minutes of instruction daily. 


Inclusion of Program Components. During small-group instruction and individual work time 
students were expected to use 3 primary program components on most days: the 44Book, the 
Decodable Digest, and the System 44 paperback book and audiobook library. In addition, on 
some days students used letter tiles and SAM practice worksheets or teachers used the 
System 44 flip chart to instruct small groups. Exhibit 5 summarizes the materials used during 
the observed teacher-led small-group instruction, and Exhibit 6 summarizes the materials used 
during the observed independent work time. In most cases half of the class used the computer 
software while the other half received small-group instruction or worked independently, and 
midway through the class period the groups changed places. 


During teacher-led small-group activities, observers most frequently noted the use of the 
System 44 flip chart, the Decodable Digest, and System 44 paperback books. While students 
worked independently, observers most frequently noted the use of System 44 paperback books, 
System 44 audiobooks, and SAM worksheets. The mean number of students using each type of 
material declined over time for System 44 audiobooks and SAM worksheets but remained about 
the same for System 44 paperback books. Level of student engagement showed no consistent 
pattern across the 3 observation periods. 


Exhibit 5 
Use of Instructional Materials During Teacher-Led Small-Group Activities 


Total Number of 


Times Observed Mean Level of Use 

Instructional Item Fall Winter Spring Fall Winter Spring 
Decodable Digest 4 1 6 1.5 2.0 1.5 
44Book 4 1 2 1.8 1.0 2.0 
Teaching resources for 1 2 0 2.0 1.0 — 
System 44 library 

Letter tiles 2 2 2 2.0 2.0 1.0 
System 44 flip chart 5 8 15 2.0 1.5 1.5 
Sound and Articulation DVD 0 0 0 — — — 
SAM resources/worksheets 5 1 1 1.8 2.0 2.0 
System 44 library books 4 2 3 1.8 2.0 2.0 
SAM reports 0 1 il — 2.0 2.0 


Note. Classrooms observed: 20. Total number of times observed is out of 25 rotations in fall, 29 in 
winter, and 45 in spring. Rating scale for level of use: 0 = none; 1 = partially; 2 = fully. Dashes indicate 
that the item was not observed. 
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Exhibit 6 
Use of Instructional Materials in Independent Work 


Total Number of Mean Number of Mean Level of 

Times Observed Students Using Student Engagement 
Type of Material F W Ss F W Ss F W Ss 
System 44 paperback 8 6 14 4.5 4.3 4.3 3.5 2.4 3.3 
books 
System 44 audiobooks 6 8 7 5.0 3.2 3.0 3.3 3.1 3.3 
44Book 2 2 5 2.5 3.0 2.4 4.0 3.0 2.8 
Decodable Digest 1 il 5 4.0 1.0 3.4 2.0 4.0 2.7 
SAM resources/ 2 6 2 5.0 4.0 3.0 2.5 2.2 3.0 


worksheets 

Note. Classrooms observed: 20. Total number of times observed is out of 25 rotations in fall, 29 in winter, and 45 in 
spring. F = fall, W = winter, S = spring. Rating scale for level of student engagement: 0 = Not at all engaged/none 
engaged; 1 = partially engaged/some engaged; 2 = fully engaged/all engaged. 


Instructional Management and Delivery. Each System 44 class observed was rated on 6 
aspects of instructional management pertaining to the teacher’s ability to maintain a positive 
learning environment, monitor students, keep students on task, provide smooth transitions 
between rotations, use flexible student groups, and differentiate support based on student 
needs. Each class was also rated on 5 aspects of instructional delivery: pacing, appropriateness 
of the delivery for the student skill level, teacher preparation, active engagement of the students, 
and overall lesson execution. Exhibit 7 summarizes the mean ratings for instructional 
management and delivery across the 3 observation time points. Overall, the observers rated 
instructional management highest in fall 2011 and rated instructional delivery highest in winter 
2012. 


Exhibit 7 also provides mean ratings of the 11 items that compose the 2 scales. The aspects of 
instructional delivery that received the lowest ratings were “appear well prepared for the lesson” 
and “execute the lesson well.” The aspects of instructional management that received the 
lowest ratings were “differentiate support based on students’ needs” and “use flexible groupings 
for students based on instructional needs.” The ratings on these instructional management 
aspects were lowest in fall 2011; ratings on instructional delivery were lowest in spring 2012. 
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Exhibit 7 
Ratings of Instructional Management and Delivery 


Fall Winter Spring 

Scale M SD M SD M SD 
Instructional Management 3.1 0.99 28 1.03 2.8 1.25 
Instructional Delivery 3.0 1.20 34 1.07 2.7 ~—-1.10 
Instructional Management 

Maintain positive learning environment 2.3 082 2.6 0.52 2.5 1.71 

Monitor students 24 083 2.6 0.70 2.6 1.06 

Keep students on task 2.7 062 2.6 0.70 2.3 0.84 

Ensure smooth transitions between rotations 2.2 107 2.2 0.79 2.3 0.77 

Use flexible groups for students based on 2.0 1.12 1.4 1.42 1.4 0.70 

instructional needs 

Differentiate support based on students’ needs 1.6 1.16 1.3 1.34 1.7 0.96 
Instructional Delivery 

Maintain an appropriate pace 24 0.77 26 0.52 2.1 0.78 

Deliver lessons appropriate for skill levels of students 2.3 065 2.5 0.71 2.0 0.71 

Appear well prepared for lesson 2.4 1.01 2.5 0.53 1.8 TAS 

Actively tries to engage students 24 0.77 26 0.70 2.4 0.70 

Executed the lesson well 2.2 1.03 2.3 0.82 1.8 0.90 


Note. Rating scale: 0 = not at all; 1 = to a small extent, 2 = to a moderate extent; 3 = definitely. Number of 
classroom observations = 20. 


Fidelity of Implementation: Teacher Self-Report 


The spring 2012 teacher survey respondents reported that students used the System 44 
software from 15 to 45 minutes daily (average across classrooms: 21.8 minutes). The number of 
computers available in each classroom ranged from 1 to 12 (average 5.4). Exhibit 8 shows the 
reported frequency of use for a range of System 44 components. The System 44 flip chart, 
44Book, and Decodable Digest were used most frequently. 


At the conclusion of the System 44 training in September 2011 and again in spring 2012 the 
participating teachers were asked to respond to 10 questions designed to test their 
understanding of the System 44 program components. On average, the teachers answered 
65% of the items correctly in fall 2011 and 76% of the items correctly in spring 2012. This 
improvement was statistically significant (p < .05). 
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Exhibit 8 
Reported Use of System 44 Components 


Mean 
Frequency 

System 44 Component Rating SD 

Whole-class instruction in System 44 4.70 0.47 
Teaching Guide (bound copy) 4.60 0.60 
44Book or Decodable Digest 4.20 0.83 
SAM reports 3.55 0.83 
Small-group instruction: SMART lessons 4.00 0.86 
Individual student support (outside of small group) 4.10 0.72 
SAM worksheets 3.21 1.27 
SAM book expert (to identify appropriate books) 2.11 1.20 
Flip chart 4.37 0.68 
Letter tiles 2.32 1.16 
System 44 audiobooks 3.65 1.39 
Reading Counts quizzes 3.26 119 
Sound and Articulation DVD 1.85 0.75 
Conference guides (for System 44 library books) 2.40 119 
Scholastic Red routines 2.95 1.19 
Materials not part of System 44 1.55 0.94 


Note. Frequency ratings 1 = rarely or never; 3 = once a week; 5 = every day. Number of 
respondents = 20. 


The teachers also rated their understanding of 5 key System 44 program components in the fall 
and spring (see Exhibit 9). Overall, the teachers reported a greater understanding of how to 
implement the program in spring 2012 than they had in fall 2011, but the differences were 
statistically significant only for the item “I understand when to use the Decodable Digest, the 
44Book, and System 44 library books for student practice.” In addition, the teachers’ responses 
to the questions about understanding of key program components varied more in the spring 
than in the fall. 


Instructional Grouping. Interviews with System 44 teachers conducted in spring 2012 
indicated that 77% used the SAM Differentiated Instruction Report to group students for 
small-group instruction. However, because some Class sizes were very small a few teachers 
had students use the computer software as a group and then receive individual instruction from 
the teacher or work independently. One teacher reported grouping students based on who 
worked well together, and another teacher reported keeping students who were on medication 
together. 
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Exhibit 9 
Understanding of Key System 44 Program Components 


Fall Spring 
Item M SD M SD 
| understand... 
How the System 44 software works to individualize 3.39 0.92 3.89 1.37 
instruction and practice for students. 
What to do during small-group instruction. 3.78 0.88 4.11 1.28 
How to use SAM to group students for small-group 3.67 0.97 4.06 1.26 
instruction. 
When to use the Decodable Digest, the 44Book, 3.28" 0.96 4.06* 1.21 
and System 44 library books for student practice. 
How to monitor student progress in System 44. 3.78 0.81 4.06 1.26 


Note. Rating scale: 1 = strongly disagree; 5 = strongly agree. Number of respondents = 18. 
*p<.05. 


Whole-Group Instruction. The teachers varied widely in terms of how frequently they 
instructed students as a whole group. Most of the 20 teachers (70%) used whole-group 
instruction every day; the remaining 30% said they did so 2 or 3 times a week. 


SAM Reports. In spring 2012 the teacher interviewees were shown 6 SAM reports and asked 
to indicate how often they used each report. About two-thirds of the teachers used the 
Differentiated Instruction Report to identify topics for small-group instruction, and 61% used the 
Software Performance Report, the Reading Progress Report, and the Response to Intervention 
Reports ranging from weekly to every 2 weeks. Least frequently used were the Student Mastery 
Report and the Parent Report. Several teachers commented that these reports duplicated 
information that could be found on other reports. 


Teacher Perceptions of the System 44 Program 


The fall 2011 the teacher survey asked respondents to rate the effectiveness of the reading 
intervention they had used the previous year (2009-2010) and their expectations regarding the 
effectiveness of System 44. The spring 2012 survey asked respondents to rate their perceptions 
of the effectiveness of System 44. Exhibit 10 provides the mean ratings for the 3 time points. 
Overall, teachers expected System 44 to be more effective than their prior year’s program in 5 
basic reading skill areas, and most of these expectations were borne out by their ratings of 
System 44 in spring 2012. The exception was comprehension—a skill area in which the 
teachers expected System 44 to be more effective than their previous program but on which 
they rated System 44 lower than their previous reading program at the end of the school year. 
The differences between the perceived effectiveness of the prior program and the System 44 
program with respect to teaching phonemic awareness and phonics were statistically significant. 
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Exhibit 10 
Perceived Effectiveness of Reading Instruction 


Perceived Expected Perceived 
Effectiveness of Effectiveness of Effectiveness of 

Prior Program System 44 System 44 

(2010-2011) (Fall 2011) (Spring 2012) 

Reading Skill Area M SD M SD M SD 
Phonemic awareness 2.88** 1.11 4.00 1.41 4.06** 1.51 
Phonics 2.88* 1.27 4.06 1.39 4.11* 1.53 
Fluency 3.18 0.95 4.00 1.37 3.44 1.42 
Vocabulary 3.59 0.80 4.17 1.15 3.61 1.38 
Comprehension 3.71 0.85 4.17 1.15 3.39 1.38 


Note. Rating scale from 1 (lowest) to 5 (highest). Number of respondents = 18. 
*p<.05, **p<.01. 


The teachers also responded to questions regarding the reading instructional practices they 
used with struggling readers both prior to and with System 44. The teachers indicated whether 
each practice listed was central to their reading instruction, a small part of their reading 
instruction, or not part of their reading instruction. The control group teachers responded to the 
same questions regarding the 2011-2012 school year. Exhibit 11 lists the practices for which 
the percentage of teachers reporting that the practice was central to their instruction changed by 
at least 25%. Appendix C presents the results for all of the reading instructional practices. 


Of the 17 instructional practices whose importance changed at least 25%, 8 reflected change in 
the System 44 teachers’ instruction between 2010-2011 and 2011-2012 and 13 reflected 
differences between the System 44 teachers and the control group teachers in 2011-2012. The 
System 44 teachers were more likely than the control group teachers to work one-to-one with 
students, work with small groups of students, group students based on skill levels, group 
students based on need for additional instruction in specific skills, use reading software, teach 
specific strategies for decoding unfamiliar words, teach decoding/phonics skills while reading 
stories, have students practice reading high frequency words for automaticity, and have 
students use knowledge of root words, prefixes, and suffixes to decode new words. The 
System 44 teachers were less likely than the control group teachers to develop reading skills 
using science and social studies texts, use teacher-made materials, use diagnostic tests to 
identify students who need reading intervention services, or have students answer questions in 
writing after reading stories. Compared to 2010-2011, in 2011-2012 the System 44 teachers 
were more likely to provide time for students to practice skills on their own, teach whole-class 
reading lessons, use reading software, teach specific strategies for decoding unfamiliar words, 
and have students use knowledge of root words, prefixes, and suffixes to decode new words. 
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Exhibit 11 
Practices Reported as Central to Reading Instruction 


System 44 System 44 Control 
Teachers: Teachers: Teachers: 
Instructional Practice Prior Year 2012-2012 2011-2012 
Provide time in reading block for students to practice 41% 85% 62% 
skills on their own. 
Develop reading skills using science and social studies 35% 25% 62% 
texts. 
Teach whole-class reading lessons. 41% 75% 64% 
Work one-to-one with students on reading. 59% 75% 29% 
Work with small groups of students. 82% 85% 59% 
Group students based on skill levels. 59% 85% 46% 
Group students based on need for additional instruction 65% 70% 45% 
in specific, targeted skills. 
Use supplementary reading materials. 82% 50% 72% 
Use separate intervention materials for some students. 76% 35% 55% 
Use reading software/technology. 29% 70% 45% 
Use teacher-made materials. 35% 15% 41% 
Use diagnostic tests to identify students who need 82% 44% 69% 
reading intervention services. 
Teach specific strategies for decoding unfamiliar 65% 90% 38% 
words. 
Teach decoding/phonics skills while reading stories. 71% 95% 41% 
Students practice reading high frequency words for 71% 75% 34% 
automaticity. 
Students use knowledge of root words, prefixes, and 59% 85% 55% 
suffixes to decode new words. 
Students answer questions in writing after reading 53% 50% 76% 


stories. 


Note. System 44 teachers n = 18. Control group teachers n = 29. 


In spring 2012 the System 44 teachers were asked to discuss the aspects of the program that 
they liked best. The teachers mentioned several software-related features: being able to fast 
track students through skills they had already mastered, the software’s ability to individualize 
instruction according to students’ needs and to revisit skills that students had not yet mastered, 
and the sense of accomplishment and empowerment students gained as they progressed 
through the software. The teachers also liked the SAM reports they could use to track student 
progress and group students for teacher-led instruction on specific skills, the variety of 

System 44 library books, the high-interest Sprint books, the Decodable Digest and the 44Book, 
and other program materials such as audiobooks and the System 44 Teaching Guide. 
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Factors That Facilitated Implementation 


The 18 System 44 teachers and 2 district staff who were interviewed for the evaluation identified 
2 primary factors that facilitated program implementation: professional development and 
coaching from Scholastic staff and the System 44 materials. 


Professional Development. Scholastic staff provided an initial training in September 2011, 
follow-up training in November, and coaching support monthly from December through May 
2012. 


System 44 Materials. The SAM reports enabled the teachers to track student progress and 
identify needs. In addition, the software identified the specific skills each student needed to 
improve, implemented the System 44 program fidelity, and provided instruction to every student 
at his or her skill level. 


Barriers to Implementation 


The teachers and district staff identified 3 primary barriers to implementation: problems with 
technology, problems with student behavior, and the difficulties inherent in learning how to teach 
a new program. Other challenges included scheduling, the disruption caused by transitioning 
between activities during the class period, minor issues related to program materials, and the 
inconveniences related to participating in the evaluation. 


Technology Problems. Teachers faced many technology problems during the school year that 
hindered program implementation. For example, older computers froze and shut down, and 
microphone and headphone problems were prevalent (mostly due to wear and tear). One 
teacher had only one adequate computer in the classroom and students had to come in ona 
staggered schedule to receive individualized System 44 instruction. 


Problems With Student Behavior. Teachers at several schools reported difficulties managing 
student behavior when they had a large number of special education students, many of whom 
had behavior issues. 


Learning to Teach System 44. Several teachers commented learning to teach using all of the 
System 44 components was overwhelming, although many praised the well-organized Teaching 
Guide. Some teachers suggested that more training on how to use SAM report data to inform 
instruction would have been helpful. 


Scheduling Issues. Scheduling concerns included the late start of the program in the school 
year (late October or early November 2011) and the resulting disruption of established student 
schedules and issues related to removing students from elective classes (e.g., art, music, and 
physical education). In addition, some classes served students with such a wide range of skills 
that grouping students for small-group instruction was difficult. One school did not have an 
adequate space to implement System 44 and 8 to 10 students and a teacher were crammed 
into a former storage closet. 


Disruption Caused by Transitioning Between Activities. A few teachers noted that the 
transitions between computer work, small-group instruction, and individual work were disruptive. 
The requirement that students spend approximately 20 minutes using the software each day 
exacerbated the problem—some teachers wanted students to be able to move from one activity 
to the next at natural breaking points such as after completing a lesson on the computer, a 
library book, or a worksheet. 


Perceived Shortcomings of the System 44 Materials. Some teachers believed that 
System 44 could be improved by providing more writing activities. In addition, some teachers 
reported that they were not sufficiently familiar with the software to assist students when they 
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asked for help, and one teacher commented that the computer instruction was too repetitive. 
One teacher said that her students did not like the Decodable Digest. 


Inconveniences Related to Participating in the Evaluation. Some principals expressed 
concern that the teachers did not have any input regarding student selection for System 44, 
whereas others felt frustrated that they could not serve all the students who might have 
benefitted from System 44. A few teachers noted that the testing window was different for 
System 44 and Read 180, which created some confusion. 
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Program Impact Findings 


Effects of System 44 on Student Outcomes 


RMC Research used multilevel models to estimate the impact of System 44 on spring 2012 
student outcome scores while controlling for fall 2011 scores, special education status (learning 
disabled/not special education), ethnicity (African American/non-African American), English 
language learner status (English language learner/non-English language learner), and school 
level (elementary/middle). The same analytic model was run for each of 4 individually 
administered standardized tests: TOSREC, CTOPP Elision, TOWRE Sight Word Efficiency, and 
TOWRE Phonetic Decoding Efficiency. Subsequent analyses examined program impacts on 
SRI and SPI scores. Appendix D presents the fixed and random effects and intraclass 
correlations for each model in Exhibits D1 through D11. 


Individual Test Outcomes 


The data analysis revealed positive results for the overall sample. The treatment group students 
performed significantly better than the control group students on 2 of the 4 individual tests: 
CTOPP Elision and TOWRE Sight Word Efficiency. Exhibit 12 presents the impact estimates for 
the overall sample. 


Exhibit 12 
Individual Test Impact Estimates 


Estimated Impact 


Two-tailed One-tailed 


Test n Impact(B) SE _ Effect Size p p 

TOSREC 315 -0.41 1.25 -0.03 745 373 
CTOPP Elision 317 1.45 0.45 0.27 .002 001 
TOWRE Sight Word Efficiency 316 2.69 0.96 0.16 .006 .003 
TOWRE Phonetic Decoding Efficiency 315 1.84 1.35 -0.09 .173 .087 


The standardized effect sizes calculated using Glass’s A method (with the impact estimate as 
the numerator and the control group standard deviation as the denominator) ranged from -0.03 
on the TOSREC to 0.27 on the CTOPP Elision. The effect sizes on the CTOPP Elision and 
TOWRE Sight Word Efficiency tests were 0.27 and 0.16, respectively. Exhibit 13 displays 
descriptive statistics (means, standard deviations, and range) for the overall sample. 
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Exhibit 13 
Individual Test Descriptive Statistics 


Treatment Control 

Test M SD Range M SD Range 
TOSREC? 

Pretest 14.79 13.56 1-64 15.99 13.03 1-47 

Posttest 18.05 13.09 1-58 18.81 14.06 1-56 
CTOPP Elision? 

Pretest 18.09 5.83 1-33 18.80 5.26 9-33 

Posttest 19.35 6.11 2-34 18.47 5.43 9-33 
TOWRE SWE” 

Pretest 51.14 15.14 1-80 53.85 14.14 5-80 

Posttest 57.71 14.33 1-82 58.02 13.67 10-84 
TOWRE PDE” 

Pretest 20.40 17.09 2-56 21.82 1s 2-54 

Posttest 23.71 11.65 2-64 24.05 11.93 3-55 


Note. SWE = Sight Word Efficiency. PDE = Phonetic Decoding Efficiency. Treatment total n = 155. 
Control total n = 162. 

“Assessment analyzed using normal curve equivalent scores. ’Assessment analyzed using raw 
scores. 


SRI and SP/ Outcomes 


Multilevel modeling on SRI and SPI outcomes also showed positive results for the overall 
sample. The impact was significantly greater for the treatment group students than the control 
group students on the SRI, B = 80.82, p = .000. The treatment group students performed 
similarly to the control group students on 2 of the SPI accuracy tests and 1 fluency test: SPI 
Letter Name Accuracy, SPI Sight Word Accuracy, and SP! Nonsense Word Fluency. However, 
the treatment group students performed significantly better than the control group students on 
SPI Nonsense Word Accuracy, B = 1.61, p = .000, and SPI Overall Accuracy, B = 1.43, p = .035. 
Treatment group students also performed marginally better than the control group students on 
SPI Sight Word Fluency, B = 0.85, p = .065, and SPI Overall Fluency, B = 1.35, p= .091. 

Exhibit 14 presents the impact estimates for the overall sample. 
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Exhibit 14 


SRI and SPI Impact Estimates 


Test n 

SRI 287 
SPI Letter Name Accuracy 279 
SPI Sight Word Accuracy 279 
SPI Sight Word Fluency 279 
SPI Nonsense Word Accuracy 279 
SPI Nonsense Word Fluency 279 
SPI Overall Accuracy 279 
SPI Overall Fluency 279 


The standardized effect sizes calculated using Glass’s A method ranged from -0.05 to 0.32 for 
SRI and SPI tests. The effect size on the SRI was 0.32. The effect sizes on tests of fluency— 


Impact (B) 
80.82 
0.03 
-0.25 
0.85 
1.61 
0.42 
1.43 
1.35 


Estimated Impact 


SE 
18.73 
0.06 
0.39 
0.50 
0.43 
0.47 
0.67 
0.80 


Effect Size 
0.32 
0.06 
-0.05 
0.17 
0.32 
0.10 
0.16 
0.16 


Two-tailed One-tailed 


p 
.000 
.653 
525 
.065 
.000 
372 
.035 
.091 


p 


.000 
327 
.263 
.033 
.000 
.186 
.018 
046 


SPI Sight Word Fluency, SPI Nonsense Word Fluency, and SPI Overall Fluency—were all 
nonsignificant or marginally significant. The effect sizes on the 2 significant accuracy tests—SPI 


Nonsense Word Accuracy and SPI Overall Accuracy—were 0.32 and 0.16, respectively. 


Exhibit 15 displays descriptive statistics (means, standard deviations, and range) for the overall 


sample. 
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Exhibit 15 
SRI and SPI Descriptive Statistics 


Treatment Control 

Test M SD Range M SD Range 
SRI 

Pretest 308.27 270.94 0-965 348.50 263.34 0-988 

Posttest 427.70 267.80 0-1,025 381.20 256.42 0-1,166 
SPI Letter Name Accuracy 

Pretest 10.82 0.47 8-11 10.70 1.30 1-11 

Posttest 10.76 0.48 9-11 10.72 0.54 8-11 
SPI Sight Word Accuracy 

Pretest 19.06 4.98 0-29 20.13 4.48 7-29 

Posttest 22.00 5.30 5-30 23.01 4.92 8-29 
SPI Sight Word Fluency 

Pretest 6.09 4.00 0-17 6.83 3.88 0-16 

Posttest 8.35 4.93 0-20 8.16 5.14 0-19 
SPI Nonsense Word Accuracy 

Pretest 16.12 4.36 5-28 16.37 4.56 5-25 

Posttest 20.95 4.65 8-30 19.59 5.10 7-30 
SPI Nonsense Word Fluency 

Pretest 5.10 3.30 0-13 5.04 3.11 0-12 

Posttest 6.30 4.77 0-22 6.19 4.30 0-17 
SPI Overall Accuracy 

Pretest 35.18 8.02 9-53 36.50 7.77 16-53 

Posttest 42.95 9.02 16-57 42.61 8.75 20-57 
SPI Overall Fluency 

Pretest 11.18 6.32 0-26 11.87 5.76 1-22 

Posttest 14.65 8.62 0-40 14.35 8.40 1-35 


Note. SRI Treatment total n = 147. Control total n = 140. SPI Treatment total n = 141. Control total n = 138. 


Effects of System 44 on Learning Disabled Students 


The hierarchical linear models conducted for each of the 4 individual reading test outcomes and 
the SRI and SPI outcomes consistently revealed main effects for learning disability (see 
Appendix D). Specifically, learning disabled students scored significantly lower than 
non-learning disabled students on all tests except for the SP! Letter Name Accuracy test. 
Additional hierarchical models were conducted to determine the impact of System 44 on the 
subsample of learning disabled students. Exhibit 16 presents the individual test impact 
estimates for the learning disabled sample, and Exhibit 17 displays descriptive statistics 
(means, standard deviations, and range) for this sample. 
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Exhibit 16 
Individual Test Impact Estimates for Learning Disabled Students 


Estimated Impact 


Two-tailed One-tailed 


Test n Impact (B) SE Effect Size p p 

TOSREC 163 0.91 2.15 0.07 .672 .336 
CTOPP Elision 165 1.82 0.61 0.36 .004 .002 
TOWRE Sight Word Efficiency 164 3.52 1.45 0.24 .017 .009 
TOWRE Phonetic Decoding Efficiency 163 -0.24 2.01 -0.01 .907 .454 


Multilevel modeling on individual tests revealed significant impacts on 2 of the 4 tests: CTOPP 
Elision and TOWRE Sight Word Efficiency. These tests were the same 2 that yielded significant 
impacts in the overall sample; however, the effect sizes for the learning disabled sample were 
larger than for the overall sample on both tests (0.36 compared to 0.27 on the CTOPP Elision 
test and 0.24 compared to 0.16 on the TOWRE Sight Word Efficiency test). 


Exhibit 17 
Individual Test Descriptive Statistics for Learning Disabled Students 
Learning Disabled Non-Learning Disabled 
Treatment Control Treatment Control 

Test M SD M SD M SD M SD 
TOSREC? 

Pretest 8.31 10.43 11.18 11.86 22.28 12.94 20.80 12.42 

Posttest 12.69 11.73 13.69 12.80 24.25 11.86 24.75 15.80 
CTOPP Elision” 

Pretest 16.56 5.64 18.25 5.27 19.90 5.56 19.35 5.22 

Posttest 17.95 6.18 17.53 5.12 21.01 5.63 19.41 5.59 
TOWRE SWE” 

Pretest 12.55 11.97 14.21 13.23 24.68 13.84 27.02 13.33 

Posttest 19.38 15.90 17.72 14.83 33.50 14.09 33.52 14.05 
TOWRE PDE” 

Pretest 14.25 16.32 13.70 14.21 21.63 17.12 25.41 17.49 

Posttest 17.11 VAAL 17.43 18.29 30.09 19.83 29.65 18.83 


Note. SWE = Sight Word Efficiency. PDE = Phonetic Decoding Efficiency. Learning disabled total n = 165. 
Non-learning disabled total n = 152. Both treatment and control included. 
“Assessment analyzed using normal curve equivalent scores. "Assessment analyzed using raw scores. 


The same hierarchical linear model was used to measure the impact of System 44 on SRI and 
SPI tests in the learning disabled sample. Exhibit 18 presents the SRI and SPI test impact 
estimates for the learning disabled sample, and Exhibit 19 displays descriptive statistics 
(means, standard deviations, and range) for this sample. 
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Exhibit 18 
SRI and SPI Test Impact Estimates for Learning Disabled Students 


Estimated Impact 


Two-tailed One-tailed 


Test n Impact (f) SE Effect Size p p 

SRI 150 93.40 26.07 0.34 .001 .001 
SPI Letter Name Accuracy 145 0.09 0.09 0.15 336 -168 
SPI Sight Word Accuracy 145 -0.13 0.58 -0.02 .829 415 
SPI Sight Word Fluency 145 1.27 0.53 0.28 .019 .010 
SPI Nonsense Word Accuracy 145 1.84 0.62 0.36 .004 .002 
SPI Nonsense Word Fluency 145 0.11 0.62 0.03 .857 429 
SPI Overall Accuracy 145 1.88 1.01 0.21 .064 .032 
SPI Overall Fluency 145 1.47 1.05 0.19 164 .082 


Similar to the results on individual tests, the results of these analyses were consistent with 
impacts observed in the overall sample. Specifically, the impact was significantly greater for the 
treatment group students than the control group students on the SRI, 6B = 93.40, p = .001, and 
on SPI Nonsense Word Accuracy, B = 1.84, p = .004. In addition, the learning disabled 
treatment group performed significantly better than the learning disabled control group students 
on SPI Sight Word Fluency, 6B = 1.27, p = .019, a test that was only marginally significant in the 
overall sample. 


To determine whether the effects of System 44 were significantly greater for learning disabled 
students than non-special education students—that is, did System 44 have a greater impact on 
learning disabled students than on non-special education students?—the moderating effects of 
special education status on treatment were analyzed. The results revealed no significant 
moderating effects. Similar tests of moderating effects were conducted for SRI and SPI 
outcomes and again no moderating effects of System 44 and learning disabled status were 
found for any of the SRI and SPI outcomes. Thus although effects on learning disabled students 
were slightly stronger, the impact of System 44 was statistically equivalent for learning disabled 
and non-special education students. 
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Exhibit 19 
SRI and SPI Descriptive Statistics for Learning Disabled Students 


Learning Disabled Non-Special Education 
Treatment Control Treatment Control 
Test M SD M SD M SD M SD 
SRI? 
Pretest 200.88 249.30 284.61 263.94 436.51 239.43 412.39 248.50 
Posttest 323.65 265.34 299.53 274.18 551.94 213.48 462.87 209.16 
SPI Letter Name 
Accuracy” 
Pretest 10.82 0.48 10.82 0.87 10.81 0.47 10.58 1.64 
Posttest 10.78 0.48 10.70 0.60 10.73 0.48 10.75 0.47 
SPI Sight Word 
Accuracy” 
Pretest 16.85 5.04 18.25 4.48 21.49 3.61 22.12 3.55 
Posttest 19.88 5.70 21.20 5.34 24.34 3.61 24.94 3.55 
SPI Sight Word 
Fluency” 
Pretest 4.78 3.17 5.69 3.66 7.52 4.35 8.04 3.78 
Posttest 6.61 3.94 6.18 4.59 10.27 5.21 10.25 4.88 
SPI Nonsense Word 
Accuracy” 
Pretest 14.68 4.25 15.11 4.14 17.72 3.92 17.70 4.63 
Posttest 19.50 5.13 18.04 5.11 22.55 3.43 21.24 4.58 
SPI Nonsense Word 
Fluency” 
Pretest 4.50 3.22 4.70 3.17 5.76 3.28 5.39 3.04 
Posttest 5.14 4.26 5.35 4.05 7.58 5.00 7.07 4.41 
SPI Overall Accuracy” 
Pretest 31.53 8.00 33.37 7.32 39.21 5.84 39.82 6.84 
Posttest 39.38 9.81 39.24 8.93 46.90 6.00 46.18 7.01 
SPI Overall Fluency” 
Pretest 9.28 5.73 10.39 5.64 13.28 6.33 13.43 5.49 
Posttest 11.74 6.97 11.54 7.62 17.85 9.18 17.33 8.21 


*SRI learning disabled total n = 150. SRI non-learning disabled total n = 137. SPI learning disabled total n = 145. 
SPI non-learning disabled total n = 134. 
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Effects of System 44 on Other Subgroups of Students 


Although the most consistent demographic subgroup main effect observed across tests was 
among learning disabled students, the main effect of school level was significant for the 
TOSREC, on which elementary school students scored higher than middle school students, and 
the CTOPP Elision, TOWRE Decoding Efficiency, SP! Nonsense Word Accuracy, and SPI 
Nonsense Word Fluency tests, on which middle school students scored higher than elementary 
school students. Results also showed main effects of ethnicity and English language learner 
status. Specifically, on the CTOPP Elision and SP! Nonsense Word Accuracy tests, non-African 
American students scored higher than African American students. On the CTOPP Elision, 
non-English language learner students scored higher than English language learner students, 
and on the SPI Overall Fluency test, English language learner students scored higher than 
non-English language learner students. Exhibits E1 through E7 in Appendix E present mean 
pretest, posttest, and gain scores on individual reading tests, and on SRI and SPI Overall 
Accuracy and Fluency outcomes for additional subgroups of students. 


RMC Research subsequently examined the moderating effects of demographic characteristics 
on System 44. The results showed that ethnicity did not moderate treatment effects—that is, the 
effects of System 44 were similar for African American and non-African American students. 
School level did, however, moderate the effects of System 44 on several tests: TOSREC, SRI, 
SPI Sight Word Accuracy, SPI Decoding Accuracy, and SPI Overall Accuracy. Exhibits 20 and 
21 present the impact estimates for elementary schools and middle schools. 


Exhibit 20 
Individual Test Impact Estimates by School Level 
Estimated Impact 


Two-tailed One-tailed 


Test n Impact (B) SE Effect Size p p 
Elementary 
TOSREC 171 -2.18 1.65 -0.15 .187 .094 
CTOPP Elision 172 1.34 0.59 0.26 025 .013 
TOWRE Sight Word Efficiency 171 2.30 1.28 0.17 .075 .038 
TOWRE Phonetic Decoding Efficiency 171 1.06 1.54 0.09 491 .246 
Middle 
TOSREC 144 2.75 1.49 0.20 .066 .033 
CTOPP Elision 145 1.72 0.64 0.30 .008 .004 
TOWRE Sight Word Efficiency 145 3.33 1.27 0.24 .010 005 
TOWRE Phonetic Decoding Efficiency 144 2.27 1.68 0.19 179 .090 


Specifically, the results showed that the System 44 and control group students performed 
similarly on the TOSREC at the elementary school level, but the System 44 students did 
marginally better than the control group students at the middle school level, p = .066. In the 
learning disabled sample, moderating effects of school level on treatment were also present, 
whereby System 44 students in the learning disabled sample scored significantly higher than 
the control group students on the TOSREC, p = .007 (results not shown). 
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Exhibit 21 
SRI and SPI Test Impact Estimates by School Level 


Estimated Impact 


Two-tailed One-tailed 


Test n Impact (B) SE Effect Size p p 

Elementary 
SRI 172 30.14 24.21 0.13 215 .108 
SPI Letter Name Accuracy 159 0.11 0.09 0.18 .213 .107 
SPI Sight Word Accuracy 159 -0.85 0.54 -0.16 119 .060 
SPI Sight Word Fluency 159 0.94 0.63 0.18 135 .068 
SPI Nonsense Word Accuracy 159 0.52 0.58 0.10 379 .190 
SPI Nonsense Word Fluency 159 0.42 0.62 0.10 .503 .252 
SPI Overall Accuracy 159 -0.34 0.92 -0.04 117 359 
SPI Overall Fluency 159 1.47 1.03 0.17 .155 .078 

Middle 
SRI 145 132.03 24.89 0.49 .000 .000 
SPI Letter Name Accuracy 120 -0.08 0.08 -0.21 .295 .148 
SPI Sight Word Accuracy 120 0.51 0.55 0.12 .359 .180 
SPI Sight Word Fluency 120 0.78 0.69 0.16 .261 131 
SPI Nonsense Word Accuracy 120 2.98 0.62 0.59 .000 .000 
SPI Nonsense Word Fluency 120 0.27 0.73 0.06 .706 .353 
SPI Overall Accuracy 120 3.61 0.97 0.46 001 001 
SPI Overall Fluency 120 1.06 1.20 0.13 .378 .189 


Appendix F provides a summary of individual, SRI, and SPI effect sizes by reading domain for 
the overall, learning disabled, and elementary and middle school samples. 


Association Between Program Implementation and Sysfem 44 Gains 


The results of the implementation analyses showed some variation in implementation across the 
participating schools. To explore the possible effects of implementation on System 44 impact, 
the evaluation team conducted nonexperimental analyses to address the following question: To 
what extent is program implementation associated with treatment impact? The data collected 
from the teacher surveys and classroom observations were used to develop 4 teacher-level 
variables (aggregated by school level) used to assess System 44 program implementation in 
relation to test outcome gains: total number of times the teacher used System 44 materials 
during small-group instruction, average number of students using System 44 paperback books 
or the Decodable Digest during independent work time, instructional management, and 
instructional delivery. 


The variable measuring the number of times the teacher used System 44 materials was a sum 
across the fall, winter, and spring classroom observations. The average number of students 
using System 44 paperback books or the Decodable Digest during independent work time was 
an average across the 3 classroom observation time points. The instructional management 
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variable comprised 2 measures (used flexible groups, used differentiated support) that were 
averaged across fall, winter, and spring. Similarly, the instructional delivery variable comprised 2 
measures (executed the lesson well, actively tried to engage students) that were averaged 
across the 3 observation points. 


Multilevel models identical to those used to measure impact—except with the 4 teacher-level 
variables (aggregated by school level) added to Level 2—were used to estimate the effects of 
implementation on the impact of the intervention on spring 2012 test outcomes. The addition of 
the teacher-level variables to the models revealed that none of the 4 variables (total number of 
times the teacher used System 44 materials during small-group instruction, average number of 
students using System 44 paperback books or the Decodable Digest during independent work 
time, instructional management, and instructional delivery) were associated with treatment 
effects for any of the tests. 


Association Between Software Usage and System 44 Gains 


Because student use of the System 44 software is an essential component of the intervention, 
the evaluation team examined how students’ System 44 software usage was related to 
improved individual test scores and SRI and SPI scores by analyzing the outcome gains in 
relation to the following factors: program exit date and total number of series topics completed. 
Exhibit G1 (in Appendix G) presents characteristics of students by exit status (i.e., early, end of 
year), and Exhibit H1 (in Appendix H) presents baseline equivalence data for these 2 groups. 
Exhibit G2 presents characteristics of students by total number of series topics completed, and 
Exhibit H2 presents baseline equivalence data for the 3 topic completion groups. Additionally, 
because a student’s initial decoding status on the SPI determines the starting series in 

System 44, Exhibit G3 presents characteristics of students by initial decoding status, and 
Exhibit H3 presents baseline equivalence data by initial decoding status. Appendix | presents 
System 44 software usage descriptive data by student subgroups. 


Program Exit Date 


The System 44 program was intended to be implemented for a full year, and the majority of 
students (92%) stayed in the program through the end of the school year (i.e., May or June 
2012). Of the 13 students who exited System 44 prior to the end of year, 1 completed the 
program in February, 6 completed the program in March, and 4 completed the program in April, 
and 2 students left the program in March and April but did not complete the program.° To 
examine differences in the characteristics of the students who completed the program early 
versus at the end of the year, the evaluation team conducted baseline equivalence tests 
between the 2 groups (see Exhibit F1). 


The students who stayed in the System 44 program through the end of the year had significantly 
lower pretest scores than the students who completed the program early on all tests except 
CTOPP Elision (which was marginally significant) and SPI Letter Name Accuracy. The only 
demographic characteristic that differed across the 2 groups was ethnicity—that is, those 
students who stayed in the program through the end of the year were significantly more likely to 
be African American than the students who completed the program early. Students who stayed 
in the program through the end of the year also were less likely to be learning disabled than 
students who exited early, but these differences were only marginally significant. 


To address whether the timing of program exit factored into students’ gains, the evaluation team 
used repeated measures analysis of variance (ANOVA) models to assess differences between 


“These 2 students who exited the program prior to the end of year without completing the program were excluded 
from analyses involving program exit date status. 
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program completion categories (i.e., early, end of year) and change in outcomes. The analyses 
revealed that early completers made significantly greater gains than end-of-year completers on 
1 test—SPI Nonsense Word Fluency, F(1, 146) = 4.76, p < .05. 


Topic Completion 


The System 44 students were expected to complete all 160 topics covered in the 25 series that 
compose the software component of the program. The analysis conducted by the evaluation 
team revealed that 21 of the 155 System 44 students (14%) completed all 160 topics (average 
number of topics completed: 77). The students were categorized by the number of topics 
completed: fewer than 40 topics (n = 52), between 40 and 100 topics (n = 57), and between 100 
and 160 topics (n = 46). Repeated measures ANOVA models were used to assess differences 
between the 3 topic completion groups and change in outcomes. 


The ANOVA results showed significant differential gains between the 3 groups on the following 
tests: TOSREC (p < .05), SP! Nonsense Word Fluency (p < .05), and SPI Overall Fluency 

(p < .05). Specifically, those students who completed fewer than 40 topics or between 40 and 
100 topics showed significantly less gain on these 3 tests than the students who completed 
between 100 and 160 topics. Additional analyses examined differences between these 3 groups 
on baseline characteristics (see Exhibit F2). Those students who completed fewer than 100 
topics had significantly lower pretest scores than the students who completed more than 100 
topics on all tests with the exception of SPI Letter Name Accuracy. Further, students who 
completed fewer than 40 topics had significantly lower pretest scores on all but 2 of these 
tests—TOSREC and CTOPP Elision—than the students who completed between 40 and 100 
topics. Tests of demographic differences between the 3 topic completion groups revealed that 
students who completed fewer than 100 topics were significantly more likely to be learning 
disabled than students who completed between 100 and 160 topics. On average, students with 
learning disabilities completed 62 topics compared to non-special education students who 
completed 95 topics. Otherwise, the 3 topic completion groups were demographically similar. 


Additional analyses examined how the 3 topic completion groups progressed on the System 44 
software. ANOVA results showed significant differences between groups on the total number of 
hours spent on the System 44 software: students who completed fewer than 40 topics spent 
significantly fewer hours on the software than students who completed between 40 and 100 
topics or between 100 and 160 topics (M = 14.60 hours, M= 23.19 hours, and M = 26.18 hours 
for the 3 groups, respectively). 


Initial Decoding Status 


A student's initial decoding status on the SPI determines the starting series in System 44 and is 
an indicator of students’ initial reading ability. Two 2 students were classified as pre-decoders, 
71 students as beginning decoders, and 82 students as developing decoders at baseline. 
Beginning decoders and developing decoders were compared on the number of total topics 
completed. Additionally, repeated measures ANOVA models were used to assess differences 
between beginning and developing decoders and change in outcomes; the 2 students classified 
as pre-decoders were excluded from these group comparison analyses. 


As would be expected, students initially classified as beginning decoders on the SPI performed 
lower than students classified as developing decoders when they began System 44. 
Specifically, beginning decoders had significantly lower pretest scores than developing 
decoders on all but 3 tests: TOSREC, CTOPP Elision, and SPI Letter Name Accuracy. 

System 44 progress also differed between the 2 groups: beginning decoders were significantly 
less likely than developing decoders to finish the program early (18% versus 49%) and 
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completed significantly fewer System 44 topics than developing decoders (M = 48.37 topics and 
M = 102.76 topics, respectively). However, there was no difference between groups in the total 
number of hours spent on the software (M = 22.13 hours and M = 20.68 hours for beginning and 
developing decoders, respectively). 


The ANOVA results showed significant differential gains between beginning and developing 
decoders on the following tests: SPI Sight Word Fluency (p < .05), and SPI Overall Fluency 

(p < .05). Specifically, beginning decoders had significantly greater gains on these 2 fluency 
tests than developing decoders. In contrast, a marginally significant (p = .051) difference existed 
on the CTOPP Elision, on which developing decoders demonstrated greater gains than 
beginning decoders. 
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Conclusion and Discussion 


The purpose of this evaluation was to assess the impact of System 44 in a low-income urban 
district that serves a large population of struggling readers and to expand the existing research 
on students with learning disabilities. Of the 368 students in Grades 4-8 who met the eligibility 
criteria for System 44, 187 were randomly assigned to the treatment group and 181 were 
randomly assigned to the control group. Nine students were deemed ineligible for the 
intervention subsequent to random assignment. The random assignment and analytic sample 
treatment and control groups were equivalent in terms of sex, eligibility for free or reduced-price 
meals, English proficiency, special education status, ethnicity, and baseline MEAP scores. Of 
the students who were eligible for the program, 155 treatment and 162 control group students 
received the allocated intervention or control group condition as planned. Half of the randomly 
assigned students (n = 185) were learning disabled (97 treatment and 88 control). 


Answers to Research Questions 
The evaluation team analyzed the data collected to address the 7 research questions: 


1. What contextual factors are involved in the implementation of System 44 (i.e., factors 
that promote or hinder successful implementation of the program)? 


As a group, the System 44 teachers had more teaching experience, were more likely to have an 
advanced degree, and were more likely to have full teaching certification than the control group 
teachers. Overall, the treatment group teachers gave the System 44 professional development 
above average ratings, with mean ratings somewhat higher in spring 2012 than in fall 2011. 
Teachers were satisfied with the quantity and quality of the training and believed that it prepared 
them to use the System 44 program in their classrooms. In response to 10 questions designed 
to test their knowledge of System 44 program components, on average the teachers correctly 
answered 65% of the questions in fall 2011 and 76% of the questions in spring 2012. 


The evaluation team assessed the fidelity of System 44 implementation through classroom 
observations and teacher self-report (Surveys and interviews). The adequacy of classroom 
setup (functioning computers, headsets, and microphones; accessible materials) improved from 
36% of the classrooms in fall 2011 to 67% in the winter and 78% in spring 2012. The proportion 
of classrooms providing at least 55 minutes of instruction each day ranged from 72% to 82% 
over time. During teacher-led small-group instruction, the observers most frequently noted the 
use of the System 44 flip chart, the Decodable Digest, and the System 44 paperback books. 
During independent work time, the observers most frequently noted the use of the System 44 
paperback books, System 44 audiobooks, and SAM worksheets. 


The observers rated instructional management highest in the fall and instructional delivery 
highest in winter 2012. Ratings on 2 key instructional management items were lowest in the fall 
(“use flexible groups for students based on instructional needs” and “differentiate support based 
on students’ needs”). At the time of the spring 2012 interviews, two thirds of the teachers 
reported that they had used the SAM Differentiated Instruction Report at least once a week to 
identify topics for small-group instruction. 


The System 44 teachers and district staff identified 3 primary barriers to implementation: 
problems with technology, problems with student behavior, and difficulties inherent in learning 
how to teach a new program. The most frequent technology problems were older computers 
that had difficulty running System 44 and problems with the student headsets and microphones. 
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2. What do teachers report to be the most valuable features of System 44? Does this 
vary for different subgroups of students? 


Prior to the start of the school year the teachers expected System 44 to be more effective than 
the reading intervention they had used the prior year; at the end of the school year the teachers 
gave System 44 significantly higher ratings in terms of its perceived effectiveness for teaching 
phonemic awareness and phonics than the prior programs. When asked which features of 
System 44 they liked best, the teachers mentioned several software-related features: being able 
to fast track students through skills they had already mastered, the software’s ability to 
individualize instruction according to students’ needs and to revisit skills that students had not 
yet mastered, and the sense of accomplishment and empowerment students gained as they 
progressed through the software. The teachers also liked the SAM reports they could use to 
track student progress and group students for teacher-led instruction on specific skills, and the 
program materials. 


3. What are the effects of System 44 on student outcomes? Specifically, how do 
changes in word-level accuracy, fluency, and reading comprehension achieved by 
System 44 students compare to changes achieved by the students in the 
services-as-usual control group? 


The findings revealed significant intervention effects on the CTOPP, TOWRE Sight Word 
Efficiency, SRI, and 2 of the SPI accuracy tests (Nonsense Word and Overall Accuracy). No 
significant effects were found on the TOSREC, TOWRE Decoding Efficiency, SPI Letter Name 
Accuracy, SPI Sight Word Accuracy or the 3 SPI Fluency tests. 


4. What are the effects of System 44 on outcomes of students with learning disabilities? 
Specifically, how do changes in outcomes achieved by System 44 students with 
learning disabilities compare to changes achieved by students with learning 
disabilities in the services-as-usual control group? 


The findings revealed similar intervention effects for the learning disabled sample on the 
CTOPP, TOWRE Sight Word Efficiency, SRI, and SP! Nonsense Word Accuracy. In addition, in 
the sample of learning disabled students, results revealed a significant effect on the SPI Sight 
Word Fluency test that was not present in the overall sample. As with the impact on the overall 
sample, no significant effects were found on the TOSREC, TOWRE Decoding Efficiency, SPI 
Letter Name Accuracy, SPI Sight Word Accuracy or the other 2 SPI Fluency tests. In addition, 
SPI Overall Accuracy was not significant in the learning disabled sample. 


5. How does System 44 differentially affect other subgroups of students? Specifically, 
how do changes in word-level accuracy, fluency, and reading comprehension 
achieved by specific subgroups of System 44 students (based on gender, ethnicity, 
economic status, English language proficiency, special education classification, and 
initial reading ability), compare to changes achieved by equivalent students who did 
not use the program? 


Analyses on subgroups of students revealed several significant differences between subgroups. 
Specifically, elementary school students scored higher than middle school students on the 
TOSREC and TOWRE Sight Word Efficiency tests, and middle school students scored higher 
than elementary school students on the CTOPP Elision, SP! Nonsense Word Accuracy, and SPI 
Nonsense Word Fluency tests. Two tests showed differences between African American and 
non-African American students, and 2 tests showed differences between English language 
learners and non-English language learners. However, the percentages of students in the 
non-African American and English language learner subsamples were relatively small, and thus 
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these findings should be interpreted with caution. Additional tests revealed that school level 
moderated the intervention effects. Specifically, several tests (TOSREC, SRI, SPI Nonsense 
Word Accuracy, and SPI Overall Accuracy) revealed significant or marginally significant effects 
for middle school students but not for elementary school students. 


6. What is the association between System 44 effects and program implementation—are 
changes in System 44 participants’ word-level accuracy, fluency, and reading 
comprehension skills associated with variation in program implementation? 


The evaluation team used data collected from teacher surveys and classroom visits to develop 
4 teacher-level variables for assessing the extent to which program implementation is 
associated with program impact: total number of times the teacher used System 44 materials 
during small-group instruction, average number of students using System 44 paperback books 
or the Decodable Digest during independent work time, instructional management, and 
instructional delivery. None of these implementation variables were associated with treatment 
effects. 


The evaluation team analyzed gains on each outcome measure in relation to program exit date 
and total number of topics completed. Students were grouped into 2 exit date categories: early 
(prior to May 2012), and end of year (May or June 2012). Not surprisingly, students who stayed 
in the System 44 program through the end of the school year had significantly lower pretest 
scores than students who completed the program. Students in the end-of-year group were also 
significantly more likely to be in special education than students in the early completion group. 
Repeated measures ANOVA analyses revealed that exit date did not factor into students’ gains 
except for one assessment—the SPI Nonsense Word Fluency subtest, on which students who 
completed the program early made significantly greater gains than students who stayed in the 
System 44 program through the end of the school year. 


Approximately 14% of the System 44 students completed all 160 topics, and the average 
number of topics completed was 77. Students were grouped by number of topics completed 
(fewer than 40 topics, between 40 and 100 topics, and between 100 and 160 topics), and 
baseline equivalence tests showed significant differences between the 3 groups on all but 1 
pretest measure. Repeated measures ANOVA analyses examined differences between 3 topic 
completion groups. Analyses revealed differential effects between groups on 3 outcomes. 
Specifically, students who completed fewer than 100 topics made significantly lower gains on 
TOSREC, SPI Nonsense Word Fluency, and SPI Overall Fluency than students in the other 
completion groups. 


7. Does progress in the System 44 software vary by students’ special education 
classification and initial reading ability? 


Students’ progress in the System 44 software varied both by their special education 
classification and their initial reading ability. Students with learning disabilities completed fewer 
topics than non-special education students (62 topics versus 95 topics). Learning disabled 
students also fast-tracked fewer topics than non-special education students (24 topics versus 49 
topics). Learning disabled students were less likely—though the results were only marginally 
significant—to finish the program early (only 27% of the students who exited early were learning 
disabled compared to 56% of students who were in the program until the end of the year). 


By the end of the year, students initially classified as beginning decoders completed an average 
of 48 topics compared to students initially classified as developing decoders who completed an 
average of 103 topics. However, time spent on System 44 software was similar between the 2 
groups. This finding is explained in part by the number of topics fast tracked by developing 
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decoders; on average, students initially classified as developing decoders fast-tracked 60 
topics, and students initially classified as beginning decoders fast-tracked only 8 topics. 


Discussion 


The evaluation findings suggest that System 44 is effective in improving reading skills for both 
struggling readers in the general population of students in Grades 4 through 8 and struggling 
readers who have learning disabilities. The evaluation findings also indicated that System 44 
had stronger effects for middle school students than for elementary school students, especially 
on measures of decoding accuracy, comprehension, and general reading achievement. One of 
the distinguishing implementation characteristics between elementary and middle schools in the 
district was class size and adherence to the System 44 model. The elementary schools that 
participated in the evaluation had far fewer students in the classroom than the middle schools. 
As a result, rotations and group sizes in elementary schools tended to be loosely structured. 
Future evaluations of System 44 might examine the extent to which group size in the classroom 
affects program impacts and whether there is an optimal group size that facilitates 
implementation of System 44 and consequently student impact. 


An important note about the evaluation is that the sample included struggling readers who were 
far below grade level in reading. Whereas phonemic awareness gains were significant and sight 
word fluency gains were marginally significant at the elementary school level, gains on more 
advanced reading skills including phonics, reading comprehension, and general reading 
achievement were significant at the middle school. The majority of middle school students 
began System 44 as developing decoders, whereas the majority of elementary school students 
began System 44 as beginning decoders. Thus differential effects of System 44 on school level 
are likely attributed to the students’ initial reading ability. Specifically, the elementary school 
students made gains on more rudimentary reading skills necessary for subsequently acquiring 
fluency and comprehension skills. The fact that on average students in higher grades completed 
more System 44 topics also supports that students’ gains were consistent with their 
advancement in the System 44 software. A follow-up study that examines continued progress of 
students in the System 44 program would provide additional support for these findings. 
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Appendix A 
Baseline Equivalence of Analytic Sample 
Treatment and Control Groups 


Treatment Group 


Exhibit A1 
Baseline Equivalence of Treatment and Control Groups 
on Outcome Measures and Demographic Characteristics 


Conirol Group 


Baseline Characteristic n ies M SD n false M SD p 

TOSREC? 154 1 14.86 13.54 162 0 15.99 13.03 .451 
CTOPP Elision® 155 0 18.09 5.83 162 0 18.80 5.26 .258 
TOWRE Sight Word Efficiency” 154 1 51.14 15.14 162 0 53.85 14.14 .101 
TOWRE Phonetic Decoding 155 0 20.41 11.05 162 0 21.69 11.23 =.310 
Efficiency 

SRI° 155 0 303.63 270.36 162 0 345.42 267.21  .167 
SPI Letter Name Accuracy® 155 0 10.82 0.46 162 0 10.67 1.43 217 
SPI Sight Word Accuracy® 155 0 19.10 4.99 162 0 20.00 4.58 .094 
SPI Sight Word Fluency® 155 0 6.17 4.11 162 0 6.76 3.96 .198 
SPI Nonsense Word Accuracy* 155 0 16.26 4.44 162 0 16.31 4.56 .921 
SPI Nonsense Word Fluency® 155 0 5.12 3.34 162 0 5.13 3.19 .971 
SPI Overall Accuracy® 155 0 35.36 8.12 162 0 36.31 7.85 .289 
SPI Overall Fluency® 155 0 11.29 6.44 162 0 11.89 5.92  .389 
Michigan Educational 133 22 557.42 123.60 141 21 565.81 129.57 .584 
Assessment Program® 

Female 155 0 39 49 162 0 .40 49 ~=.889 
Black 155 0 .80 .40 162 0 17 42 458 
Hispanic 155 0 10 .30 162 0 13 34 = 358 
FRL 155 0 95 .20 162 0 .96 19 .716 
SPED Status 155 0 54 .04 162 0 .50 04 = .457 
ELL Status 155 0 .04 .29 162 0 .06 23 ~=©.482 


*Pretest assessment was analyzed using normal curve equivalent scores that ranged from 1 to 99. 
’Assessment was analyzed using raw scores. “Assessment was analyzed using Lexile scores.“Assessment 
was analyzed using scaled scores. 
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Appendix B 
Data Collection Instruments 


System 44 Teacher Survey 


Saginaw Public Schools 


September 2011 


About This Survey 


This survey contains questions about your background and questions about the 

System 44 program that you will be implementing this school year. Your responses are 
important in helping us understand which factors contribute to the success of the program 
and what issues need to be addressed. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff to assist in planning 
for System 44 implementation. 


We expect it will take approximately 15 minutes to complete the survey. 


Thank you very much for your help. 


Prepared by: 
RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Teacher Background 


1. Are you female or male? 
O, Female 
O, Male 
2. How many years have you worked as a full-time teacher in public schools? Write in the number 


of years below. Count part of a year as one year. 

Number of years 
a. Total number of years as a teacher (do not include student teaching) 
b. Total number of years as a teacher in Saginaw Public Schools 


c. Total number of years as a teacher at your current school 


3. What is the highest degree you have obtained as of September 2011? Mark (X) only one box. 
Oj, Bachelor's (B.A., B.S., B.E., etc.) 
1, Master’s degree (M.A., M.A.T., M.B.A., M.Ed., M.S., etc.) 
O, Education specialist or certification at least one year beyond master’s level 


, Doctorate or professional degree (Ph.D., Ed.D., M.D., L.L.B., J.D., D.D.S.) 


O Other (please specify: ) 
4. Which of the following describes the teaching certificate(s) you currently hold in Michigan? 
Mark (X) all that apply. 


O. Temporary Teacher Employment Authorization 
1, Interim Teaching Certificate 
O. Provisional Certificate 


OO, Professional Education Certificate 


5: | have used System 44 in my classroom prior to this year. 
O, Yes OO, No 

6. | have used Read 180 in my classroom prior to this year. 
O, Yes O, No 
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Professional Development 


7. Have you had training or support (e.g., in-classroom coaching, online support) provided by a 
Scholastic representative or other experienced System 44 user prior to today’s training? 
Mark (X) all that apply. 


O. Have not received any training in System 44 prior to today’s training 
1, Attended a similar System 44 training held in a prior year 
CO. Received other support for System 44 from a Scholastic representative in a prior year 


1, Received other support from another experienced System 44 user in a prior year 


O. Other (please specify: ) 
8. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 training activities provided today. Mark (X) only one box per row. 
Strongly Strongly 
disagree agree 


a. The teacher training on System 44 prepared me to use Oo, 0, 0, aa OO, 
the program in my classroom. 


b. 1am pleased with the amount of System 44 Oo, EL 0, Oo, Oo, 
professional development | received. 
c. lam pleased with the quality of System 44 oo, Oo, Oo, Oo, Oo, 
professional development | received. 
Instructional Practices 
9. Please indicate which of the following teaching strategies and materials you used with your 


struggling readers last year (2010-2011). Mark (X) only one box per row. 


If you did not teach last year but taught in previous years, please describe your use of these 
teaching strategies and materials when you last taught. If this is your first year teaching, please 
skip to Question 12. 


Central to Small part of Not Part of 


my reading my reading my reading 
instruction instruction instruction 
Instruction 
a. Provide time in reading block for students to 0, Oo, as 
practice skills on their own. 
b. Provide materials for at-home practice of skills oO, Oo, a 
introduced in class. 
c. Provide extra reading instructional time for ERs 0, i 
struggling readers. 
d. Include writing opportunities in reading instruction. a, a Oo; 
e. Build spelling practice into reading instruction. 1, =P 1, 
f. Develop reading skills using science and social oO, A Fl; 


studies texts. 
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Central to Small part of Not Part of 


my reading my reading my reading 
instruction instruction instruction 
Grouping 
g. Teach whole class reading lessons. Oo, Oo 0, 
h. Work one-to-one with students on reading. 1, mp 1, 
i. Work with small groups of students. oo, ma hi 
j. Group students based on skill levels. Oo, 0, 0, 
k. Group students based on need for additional 1, mp aE 
instruction in specific, targeted skills. 
l. Group students based on mixed abilities (pairs or 1, 1 1, 
cooperative groups). 
Reading Materials 
m. Use core reading series. Oo, 0, 0, 
n. Use supplementary reading materials. 1, mp Oo, 
0. Use trade books. 0, 0, 0, 
p. Use books that are easy to decode. Oo, 0, 0, 
q. Use separate intervention materials for some 1, mp 1, 
students. 
r. Use reading software/technology. oo, mp 1, 
s. Use teacher-made materials. Fae mp 0, 
Assessments 
t. Use test results to organize instructional groups. 1, mp 1, 
u. Use informal reading inventories. 1, ea FS 0, 
v. Conduct miscue analysis, analyzing errors 1, ia 1, 
students make while reading aloud. 
w. Use tests to determine progress on skills. 1, mp 1, 
x. Use diagnostic tests to identify students who need Oo, Oo Oo 


reading intervention services. 
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10. Please indicate which of the following reading instructional activities you used with your 
struggling readers last year (2010-2011). Mark (X) only one box per row. 


If you did not teach last year but taught in previous years, please describe your use of these 
reading instructional activities when you last taught. If this is your first year teaching, please skip 
to Question 12. 


Central to Small part of Not Part of 


my reading my reading my reading 
instruction instruction instruction 
Reading Text 
a. Students reread familiar text. 1, aa Fs 1, 
b. Students confirm or revise predictions after oO, Oo, Oo, 
reading. 
c. Students generate their own questions about text oO, Oo, Eas 
material. 
d. Students identify their comprehension break-downs oO, Oo, mae 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories 0, Oo, 0, 
and informational texts. 
f. Students use graphic and semantic organizers to 1, oO, 0; 
track information. 
Work With Sounds and Words 
g. | teach specific strategies for decoding unfamiliar 0, Oo, Oo, 
words. 
h. | teach decoding/phonics skills while reading Ey im 0; 
stories. 
i. Students practice reading high frequency words for oo, Oo, 0; 
automaticity. 
j. Students use knowledge of root words, prefixes, oO, Oo, mae 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change 0, 0, [ne 
the meaning of words. 
|. Students use context clues to identify unknown oO, Oo, a 
words. 
m. | discuss new and unusual words before reading. 1, mp en 
Other Techniques 
n. Students answer questions in writing after reading oO, Oo, Oo, 
stories. 
0. Students select books from the library for ee fens mae 
independent reading. 
p. Students are given time to read on their own for o, Oo, Oo, 


enjoyment. 
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11. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


My struggling students last year received enough 


instruction and practice in the following areas to Strongly Strongly 
make sufficient gains in reading. disagree agree 
a. phonemic awareness Gy J6b- Ge iB ws 
b. phonics Ga. Ee. We, Te he 
c. fluency oo: Es. Gy sy: . Li: 
d. vocabulary Ey He. Gh “Me «Gh 
e. comprehension (ly. Ie. le 5..~ ele 


12. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


| think System 44 will be effective this Strongly Strongly 
year in helping my students with... disagree agree 
a. phonemic awareness. Ge. J. “Gh. ie. “Ble 
b. phonics. A. Ge EE. fy ok 
c. fluency. he: (Ee. Se ti. «Be 
d. vocabulary. Ey ie. SO. icy. 2: 
e. comprehension. Pe. te. Eh Wi. vile 


System 44 Concepts 

The following questions are intended to capture your current understanding of the System 44 program 
based on either prior knowledge of the program or what you learned in the training today. This 
information will help us to understand areas that the training did or did not address, and will help guide 
future training sessions. 


13. Which of the following is NOT a skill that is measured by the Scholastic Phonics Inventory? 
Mark (X) only one box. 


1, Identify sight words 
O., Recognize letter names 
Identify nonsense words 


J, Improve phonological awareness 
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14. Which of the following System 44 program components is designed to provide students with 
written practice in applying the skills they are learning in the software? Mark (X) all that apply. 


O. Decodable Digest 
O, 44Book 
1. Flip chart 
O14 SAM Practice Pages 
15: Which of the following System 44 program components is designed to provide reading fluency 
practice for students? Mark (X) all that apply. 
O). Decodable Digest 
1, System 44 Library Books 
OO. System 44 Audiobooks 


Ol, Success Strand 


16. Scholastic Reading Counts! is... Mark (X) only one box. 
CO, the System 44 adaptive software that continuously collects data on student performance. 
©. aset of quizzes that measure comprehension of each System 44 paperback book. 
Oz aset of progress monitoring tools given at the end of each System 44 series. 
O, aset of motivational videos that students can play when they have successfully completed 
a System 44 series of lessons. 
17. Which of the following is NOT a feature of System 44? Mark (X) only one box. 
O, Using SAM reports for instructional grouping 
©. Teacher-led instruction in small groups 
Library books that reinforce specific phonics skills 


1, Successful completion of every software lesson 


18. Phonemic awareness is . . . Mark (X) all that apply. 
O], the foundation for phonics instruction. 
the ability to hear and manipulate individual sounds in spoken words. 
Othe same as phonological awareness. 


1, the recognition that sentences consist of separate words. 
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19. The Scholastic Achievement Manager (SAM) can... Mark (X) all that apply. 
help teachers form instructional groups for students who need work on the same skills. 
provide detailed, diagnostic information about student strengths and weaknesses. 
©. allow the teacher to monitor students’ ongoing progress on the software. 
, provide downloadable assessments for various reading skills. 


20. Select the description from the right column that matches each of the four System 44 Zones 
listed on the left and enter the matched number in the box: 


a. Fluency Zone (3) 1 Explicit instruction in the 44 sounds of the English 
language and the letters that make each sound 

b. Spelling Zone fs) 2 Reading of decodable text with a focus on 
comprehension 

c. Word Zone [ 3 Instruction and practice in specific spelling rules 

d. Smart Zone (= 4] 4 Modeling and practice in blending sounds 


21. Select the description from the right column that matches each of the four System 44 strands of 
instruction listed on the left and enter the matched number in the box: 


a. Success f_ 3) 1 Builds automatic recognition of high frequency 
words 

b. Word Strategies [ 2 Read passages to build on skills and improve 
comprehension 

c. The Code [) 3 Practice in syllable strategies and word analysis 

d. Sight Words [_} 4 Direct instruction and practice in letter-sound 
correspondences 


22. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 program. Mark (X) only one box per row. 


Strongly Strongly 
| understand... disagree agree 
a. how the System 44 software works to hy Oo, Oo, oO, O, 
individualize instruction and practice for students. 

b. what to do during small group instruction. 0, oO, 0; oO, O, 

c. how to use SAM to group students for small mn Oo, 0, Oo, O. 
group instruction. 

d. when to use the Decodable Digest, 44Book, and mn mp 0, 1, O. 
System 44 library books for student practice. 

e. how to monitor student progress in System 44. mn a 1, Oo, ele 
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23. Some questions | have about System 44 are... 


RMC Research Corporation@Portland, OR System 44 Teacher Survey 


System 44 Teacher Survey 


Saginaw Public Schools 


May 2012 


About This Survey 


This survey contains questions about your background and questions about the 

System 44 program that you implemented this school year. Your responses are important 
in helping us understand which factors contribute to the success of the program and what 
issues need to be addressed. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff to assist in 
understanding System 44 implementation. 


We expect it will take approximately 15 minutes to complete the survey. 


Thank you very much for your help. 


Ws Prepared by: 

Za RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Professional Development 


de Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 training activities provided this year. Mark (X) only one box per row. 
Strongly Strongly 
disagree agree NIA 
a. The group training on System 44 prepared me mn Oo, Oo, Oo, O, Oo, 
to use the program in my classroom. 
b. The individual support from Scholastic during 0, 0, 0, Oo, fis Ea 
the year enhanced my skills in using 
System 44 in my classroom. 
c. The individual support from district staff during Oo, Oo, Gl Oo, O, Oo, 
the year enhanced my skills in using 
System 44 in my classroom. 
d. 1am pleased with the amount of System 44 0, 0, 0, [Ny 0, 0, 
professional development | received. 
e. lam pleased with the quality of System 44 oo, Oo, Oo, Oo, O, Oo, 
professional development | received. 
2. Which training or support topics were most helpful to you as you implemented System 44? 
3. What additional System 44 training or support would have been helpful? 
Implementation of System 44 Components 
4. How frequently did you use each of the following System 44 components? 
Rarely or Oncea Every 
never week day 
a. Whole class instruction in System 44 0, 0, 0, Oo, Oo. 
b. Teaching Guide (bound copy) oO, Oo, Oo, Oo, O, 
c. 44 Book or Decodable Digest ea Oo, Oo, Oo, Oo, 
d. SAM reports oO, Oo, Oo, Oo, O, 
e. Small group instruction: SMART lessons oO, Oo, El Oo, Oo, 
f. Individual student support (outside of small group) oO, Oo, Oo, Oo, Oo, 
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5: How frequently did you supplement your System 44 instruction with the following materials? 


Rarely or Oncea Every 
never week day 
a. SAM practice pages on FB OB &% OO; 
b. SAM book expert (identify appropriate books) 0, mp OO, 1, OO. 
c. Flip chart Se: “Eh. Je- Ele. Jefe 
d. Letter manipulatives [hy hy Cla. thy. he 
e. System 44 audiobooks iS; . Gis Age. Ble, “Gs: 
f. Reading Counts! quizzes Pe i. Gi 7g “Ge 
g. Sound and Articulation DVD ‘7, oF 08 co. oO. 
h. Conference Guides (for 44 Library books) Ol, te Ge ae. “Gh 
i. Scholastic Red routines cy os GB; ©) Gs: 
j. Other materials that were not System 44 Ey. I 6 G.. tie 
6. On average, how many minutes did each student use System 44 on the computer each day? 


How many computers were available in your classroom for System 44 instruction? 


Instructional Practices 
fe Please indicate which of the following teaching strategies and materials you used with your 


struggling readers during System 44 instructional time this year (2011-2012). Mark (X) only 
one box per row. 


Central to Small part of Not part of 


my reading my reading my reading 
instruction instruction instruction 
Instruction 
a. Provide time in reading block for students to oO, Oo, a 
practice skills on their own. 
b. Provide materials for at-home practice of skills oO, Oo, a 
introduced in class. 
c. Provide extra reading instructional time for oO, Oo, Oo, 
struggling readers. 
d. Include writing opportunities in reading instruction. Ele mp 1, 
e. Build spelling practice into reading instruction. 1, mp 1, 
f. Develop reading skills using science and social oo, oO, Oo, 


studies texts. 
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Central to Small part of Not part of 


my reading my reading my reading 
instruction instruction instruction 
Grouping 
g. Teach whole class reading lessons. Oo, Oo 0, 
h. Work one-to-one with students on reading. 1, mp 1, 
i. Work with small groups of students. oo, ma hi 
j. Group students based on skill levels. Oo, 0, 0, 
k. Group students based on need for additional 1, mp aE 
instruction in specific, targeted skills. 
l. Group students based on mixed abilities (pairs or 1, 1 1, 
cooperative groups). 
Reading Materials 
m. Use core reading series. Oo, 0, 0, 
n. Use supplementary reading materials. 1, mp Oo, 
0. Use trade books. 0, 0, 0, 
p. Use books that are easy to decode. Oo, 0, 0, 
q. Use separate intervention materials for some 1, mp 1, 
students. 
r. Use reading software/technology. oo, mp 1, 
s. Use teacher-made materials. Fae mp 0, 
Assessments 
t. Use test results to organize instructional groups. 1, mp 1, 
u. Use informal reading inventories. 1, ea FS 0, 
v. Conduct miscue analysis, analyzing errors 1, ia 1, 
students make while reading aloud. 
w. Use tests to determine progress on skills. 1, mp 1, 
x. Use diagnostic tests to identify students who need Oo, Oo Oo 


reading intervention services. 
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8. Please indicate which of the following reading instructional activities you used with your 
struggling readers during System 44 instructional time this year (2011-2012). Mark (X) only 
one box per row. 


Central to Small part of Not part of 


my reading my reading my reading 
instruction instruction instruction 
Reading Text 
a. Students reread familiar text. 1, mp El; 
b. Students confirm or revise predictions after oO, Oo, (a 
reading. 
c. Students generate their own questions about text ee i aE 
material. 
d. Students identify their comprehension break-downs o, Oo, 0, 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories oo, i 0; 
and informational texts. 
f. Students use graphic and semantic organizers to 1, oO, Ol, 
track information. 
Work With Sounds and Words 
g. Iteach specific strategies for decoding unfamiliar 0, 0, am 
words. 
h. | teach decoding/phonics skills while reading oo, oo, OO; 
stories. 
i. Students practice reading high frequency words for oO, mF mae 
automaticity. 
j. Students use knowledge of root words, prefixes, o, 0, 0, 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change oO, Oo, Oo, 
the meaning of words. 
|. Students use context clues to identify unknown oO, [Ns i 
words. 
m. | discuss new and unusual words before reading. Oo, 0, 0, 
Other Techniques 
n. Students answer questions in writing after reading oO, Oo, a 
stories. 
o. Students select books from the library for me Oo, al 
independent reading. 
p. Students are given time to read on their own for 1, i 0; 


enjoyment. 
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9. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


My System 44 students this year received enough 


instruction and practice in the following areas to Strongly Strongly 

make sufficient gains in reading. disagree agree 
a. phonemic awareness By WG. ge ay. «Ge 
b. phonics on, o&% OO m&% OO; 
c. fluency on, 8 8 % OO; 
d. vocabulary ly de. igs ey. ~ Ee 
e. comprehension Ea (Ee 7.. fy . Ge 


Other Reading Interventions 


10. This school year (2011-2012) did you teach struggling readers who were eligible for System 44 
but did not receive System 44 instruction (i.e., students who were in the study’s control group)? 


J, Yes (continue with Question 11) 


O. —_No (skip to Question 13) 


11. What reading instructional program(s) did you use this year with struggling readers who were 
eligible for System 44 but did not receive System 44 instruction (i.e., students who were in the 
study’s control group)? Mark (X) all that apply. 


Of Houghton-Mifflin On Explode the Code 
O» Morphographs Oj} + Rosetta Stone 

oO. Ticket to Read Oj} Language! 

Og Read Naturally O, Read 180 

Oe SRA Decoding Oo, System 44 

O, Voyager On, Other 

O, REACH O, Other 


12. What types of assistance were provided to struggling readers in your classroom who were part 
of the study’s control group this year? Mark (X) all that apply. 


©. Pull-out resource room instruction (for special education students) 
CO), In-class instructional aide (for special education students) 

O1, Pull-out Title | instruction 

O14 In-class instructional aide (for Title | students) 

O.. After school tutoring 


O; Other (please describe): 
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System 44 Concepts 

The following questions are intended to capture your current understanding of the System 44 program 
based on either experience with the program or what you learned in the training. This information will 
help us to understand areas that the training did or did not address, and will help guide future training 
sessions. 


13. Which of the following is NOT a skill that is measured by the Scholastic Phonics Inventory? 
Mark (X) only one box. 


1, Identify sight words 
O., Recognize letter names 
Identify nonsense words 
1, Improve phonological awareness 
14. Which of the following System 44 program components is designed to provide students with 
written practice in applying the skills they are learning in the software? Mark (X) all that apply. 
O. Decodable Digest 
O, 44Book 
1. Flip chart 
O14 SAM Practice Pages 
LS: Which of the following System 44 program components is designed to provide reading fluency 
practice for students? Mark (X) all that apply. 
O), Decodable Digest 
1, System 44 Library Books 
OO. System 44 Audiobooks 


Ol, Success Strand 


16. Scholastic Reading Counts! is... Mark (X) only one box. 
CO, the System 44 adaptive software that continuously collects data on student performance. 
O, aset of quizzes that measure comprehension of each System 44 paperback book. 
CO; aset of progress monitoring tools given at the end of each System 44 series. 


O, aset of motivational videos that students can play when they have successfully completed 
a System 44 series of lessons. 
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17. 


18. 


19. 


20. 


21. 


Which of the following is NOT a feature of System 44? Mark (X) only one box. 
O, Using SAM reports for instructional grouping 

©. Teacher-led instruction in small groups 

Library books that reinforce specific phonics skills 


1, Successful completion of every software lesson 


Phonemic awareness is . . . Mark (X) all that apply. 

1, the foundation for phonics instruction. 

J, the ability to hear and manipulate individual sounds in spoken words. 
CO. the same as phonological awareness. 


Ol, the recognition that sentences consist of separate words. 


The Scholastic Achievement Manager (SAM) can... Mark (X) all that apply. 

(1. help teachers form instructional groups for students who need work on the same skills. 
1, provide detailed, diagnostic information about student strengths and weaknesses. 
©. allow the teacher to monitor students’ ongoing progress on the software. 


CM, provide downloadable assessments for various reading skills. 


Select the description from the right column that matches each of the four System 44 Zones 
listed on the left and enter the matched number in the box: 


a. Fluency Zone = 1 Explicit instruction in the 44 sounds of the English 
language and the letters that make each sound 


b. Spelling Zone [ 2 Reading of decodable text with a focus on comprehension 
c. Word Zone [ 3 Instruction and practice in specific spelling rules 
d. Smart Zone (e—3] 4 Modeling and practice in blending sounds 


Select the description from the right column that matches each of the four System 44 strands of 
instruction listed on the left and enter the matched number in the box: 


a. Success = 1 Builds automatic recognition of high frequency words 


b. Word Strategies [ 2 Read passages to build on skills and improve 


comprehension 
c. The Code [ 


d. Sight Words = 


wo 


Practice in syllable strategies and word analysis 


& 


Direct instruction and practice in letter-sound 
correspondences 
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22. Please rate the extent to which you agree or disagree with the following statements regarding 
the System 44 program. Mark (X) only one box per row. 


Strongly Strongly 
| understand... disagree agree 
a. how the System 44 software works to a Be. Gs. way. Gh 
individualize instruction and practice for students. 

b. what to do during small group instruction. Ey “O- Hy “Gy: Ge 

c. how to use SAM to group students for small oo OO. OF iD, GO: 
group instruction. 

d. when to use the Decodable Digest, 44Book, and Ay dg. -Be- hy Gs 
System 44 library books for student practice. 

e. how to monitor student progress in System 44. Oe. Els (2. “y- <i 


23. Some comments | have about System 44 are... 
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Teacher Survey 


Saginaw Public Schools 


May 2012 


About This Survey 


This survey contains questions about your background and questions about your reading 
instruction this school year. Your responses are important in helping us learn more about 
reading instruction in your district. 


Please write your answers directly on the survey by checking the appropriate boxes or by 
writing your response in the space provided. Your individual responses will be kept 
confidential and will not be shared with your principal or other school or district personnel. 
Summary data from this survey will be shared with Scholastic staff as part of a System 44 
study that is taking place in your district. 


We expect it will take approximately 15 minutes to complete the survey. Please return the 
survey and accompanying consent form to us in the self-addressed stamped envelope 
we provided. Upon receipt of this survey, we will email you a $25 gift certificate to the 
Scholastic Teacher Store. 


Thank you very much for your help. 


Prepared by: 
RMC Research Corporation 
111 SW Columbia Street 
Suite 1200 

RESEARCH Portland, OR 97201 


Teacher Background 


1. Are you female or male? 
O, Female 
O, Male 
2. How many years have you worked as a full-time teacher in public schools? Write in the number 


of years below. Count part of a year as one year. 

Number of years 
a. Total number of years as a teacher (do not include student teaching) 
b. Total number of years as a teacher in Saginaw Public Schools 


c. Total number of years as a teacher at your current school 


3. What is the highest degree you have obtained as of September 2011? Mark (X) only one box. 
Oj, Bachelor's (B.A., B.S., B.E., etc.) 
1, Master’s degree (M.A., M.A.T., M.B.A., M.Ed., M.S., etc.) 
O, Education specialist or certification at least one year beyond master’s level 


, Doctorate or professional degree (Ph.D., Ed.D., M.D., L.L.B., J.D., D.D.S.) 


O Other (please specify: ) 
4. Which of the following describes the teaching certificate(s) you currently hold in Michigan? 
Mark (X) all that apply. 


O. Temporary Teacher Employment Authorization 
1, Interim Teaching Certificate 
O. Provisional Certificate 


OO, Professional Education Certificate 


5: | have used System 44 in my classroom. 
O, Yes 1, No 

6. | have used Read 180 in my classroom. 
O, Yes O, No 
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Professional Development 


7. Have you had training or support in System 44 (e.g., in-classroom coaching, group training) 
provided by a Scholastic representative or other experienced System 44 user? Mark (X) all that 
apply. 


O. Have not received any training in System 44 

O,, Attended a System 44 training 

O. Received other support for System 44 from a Scholastic representative 
CO, Received other support from another experienced System 44 user 


C). Other (please specify;  ——ss—s—C—i‘“SY 


Instructional Practices 


8. Please indicate which of the following teaching strategies and materials you used with your 
struggling readers this year (2011-2012). Mark (X) only one box per row. 


Central to Small part of Not part of 


my reading my reading my reading 
instruction instruction instruction 
Instruction 

a. Provide time in reading block for students to 0, Oo Oo; 
practice skills on their own. 

b. Provide materials for at-home practice of skills 1, mp 1, 
introduced in class. 

c. Provide extra reading instructional time for 1, mp ia 
struggling readers. 

d. Include writing opportunities in reading instruction. Oo, Oo O 

e. Build spelling practice into reading instruction. 1, mp Els 

f. Develop reading skills using science and social oO, i 1, 
studies texts. 

Grouping 

g. Teach whole class reading lessons. Oo, 0, 0, 

h. Work one-to-one with students on reading. 1, mp 1, 

i. Work with small groups of students. 0, Oo Oo 

j. Group students based on skill levels. Oo, i 0, 

k. Group students based on need for additional 1, mp 1, 
instruction in specific, targeted skills. 

|. Group students based on mixed abilities (pairs or oo, mp 1, 


cooperative groups). 
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Reading Materials 


m. Use core reading series. 

n. Use supplementary reading materials. 

o. Use trade books. 

p. Use books that are easy to decode. 

q. Use separate intervention materials for some 
students. 

r. Use reading software/technology. 

s. Use teacher-made materials. 

Assessments 

t. Use test results to organize instructional groups. 

u. Use informal reading inventories. 

v. Conduct miscue analysis, analyzing errors 
students make while reading aloud. 

w. Use tests to determine progress on skills. 

x. Use diagnostic tests to identify students who need 


reading intervention services. 


Central to 
my reading 
instruction 


Small part of Not part of 


my reading my reading 

instruction instruction 
Oo, Oo 
Oo, 0; 
Oo, 0, 
mp O, 
0, Oo, 
O. O23 
O, O; 
Oo, Oo, 
me 0; 
Oo, Oo; 
O., O; 
mp O; 
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System 44 Study Design 


9. Please indicate which of the following reading instructional activities you used with your 
struggling readers this year (2011-2012). Mark (X) only one box per row. 


Central to Small part of Not part of 


my reading my reading my reading 
instruction instruction instruction 
Reading Text 
a. Students reread familiar text. Oo, EI as 
b. Students confirm or revise predictions after oO, ms ma 
reading. 
c. Students generate their own questions about text me Oo, Oo, 
material. 
d. Students identify their comprehension break-downs o, Oo, a 
and use fix-up strategies with a partner. 
e. Students orally summarize main events in stories oO, Oo, FIs 
and informational texts. 
f. Students use graphic and semantic organizers to oO, Oo, Oo, 
track information. 
Work With Sounds and Words 
g. Iteach specific strategies for decoding unfamiliar i 0, 0, 
words. 
h. | teach decoding/phonics skills while reading mn oO, El 
stories. 
i. Students practice reading high frequency words for oO, Oo, Oo, 
automaticity. 
j. Students use knowledge of root words, prefixes, 1, Oo, fas 
and suffixes to decode new words. 
k. Students work with prefixes and suffixes to change oO, Oo, mae 
the meaning of words. 
|. Students use context clues to identify unknown 0, i 0, 
words. 
m. | discuss new and unusual words before reading. ae ea ia FS 
Other Techniques 
n. Students answer questions in writing after reading me Oo, Oo, 
stories. 
0. Students select books from the library for oO, Oo, mae 
independent reading. 
p. Students are given time to read on their own for oO, Oo, Oo, 


enjoyment. 


RMC Research Corporation@Portland, OR SYA Cue 2 ES Ue Na esiela) 


10. What reading instructional program(s) did you use this year with your struggling readers? 


Mark (X) all that apply. 

Of. Houghton-Mifflin On Explode the Code 
O» Morphographs O1 + Rosetta Stone 

O. Ticket to Read Oj} Language! 

Ou Read Naturally O,. Read 180 

Oe SRA Decoding O, System 44 

O; Voyager Om Other 

O, REACH O,, Other 


11. What types of assistance were provided to students in your classroom this year who are 
struggling with reading? Mark (X) all that apply. 


Oj, Pull-out resource room instruction (for special education students) 
Oo. In-class instructional aide (for special education students) 
OO, —— Pull-out Title | instruction 
mF In-class instructional aide (for Title | students) 
O. After school tutoring 
O, Other (please describe): 
12. Please rate the extent to which you agree or disagree with the following statements. Mark (X) 
only one box per row. 


My struggling students this year have received 


enough instruction and practice in the following Strongly Strongly 
areas to make sufficient gains in reading. disagree agree 
a. phonemic awareness Bi JG. “Ge “Ge Gs 
b. phonics or Os. Oy Dy -Ge 
c. fluency Ge "Os °- Gs “Oy 
d. vocabulary A es By ye 
e. comprehension oo Os 63 OO. 
Thank you! 
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System 44 


Saginaw Public School District 
Classroom Visit 


O71, Coulter O, Kempton O., Rouse O.; Ruben Daniels Middle 
O, Heavenrich 0, Longfellow O,, Stone 0.1, Thompson Middle 
O Herig O, Loomis O., Arthur Eddy K-8 
O7,,, Houghton Of. Merrill Park O11, Zilwaukee K-8 
O; Jerome O.. Miller 
4 eee 


111 SW Columbia Street 
Suite 1200 
RESEARCH Portland, OR 97201 
Fall Winter Spring 
2011 2012 2012 


Date: Class Visit #: 1 2 3 
Number of Students in Class During Visit: Teacher Name: 
Observer: System 44 start time: System 44 end time: Total minutes: 


Grade Levels Represented: ©, 4" OFn5" of, 6" oFf247" Of 8" 
Class Type: Oj, Pullout Class O. Replacement ELA Class (mark ail that apply.) 
Number of Computers Available for System 44: Number of Other Adults in Class: ____ 


Role/Activity of Other Adult(s): 


Teaching or organizational strategies used: 


Notes: 


Interval start time: 


Interval end time: 


Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


What Is the teacher doing, and how fully is the teacher 
implementing each activity? 


Teacher-Led Support 


What supports is the teacher using, 
and how tully” is the teacher using them? 


0), Teaching Guide* O1, Decodable Digest O©O®@ 

e Introducing @O®@ | 0. 44 Book @DO®@ 

aol e Teaching/modeling @@®@ | Oa Teaching Resources for the System 44 Library @D®@ 

> 5 e Engaging students in guided practice | ©O@ | O, Letter manipulatives @Oe@ 

os e Apply and reinforcing/reviewing @O®@ | O; Flip chart @©O® 

CO | Notes. O, Sound and articulation DVD @©O®@ 

i) = * The “I Do, We Do, You Do” components should be incorporated for full 

E 2 Neeeetne Oj, SAM Resources/Worksheets ODO® 

E ? Full use of supports involves modeling answers and guiding student 0; System ae paperback books @©O®@ 

~*= usage of lessons; monitoring student work; and providing verbal or written OO, SAM Data: Reading Progress Report OO®@ 
feedback. If a teacher assigns an activity using a material but does not ! = = : = 

provide any guidance, monitoring, or feedback, mark 0. O;, SAM Data: Differentiated Instruction Grouping Report ODO 

Om Non-System 44 materials 0) SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


System 44 computer software ( students) 


© Not at all engaged/none engaged 


0) Partially engaged/some engaged 


@ Fully engaged/all engaged 


Independent (Individual/Paired) Activity 


What are individual students or pairs 
doing during small group instruction? Mark all that apply. 


Independent (Individual/Paired) Student Support 


What supports are the students using, 


and how engaged? are students in their use of them? 


O. Writing activity using System 44 materials O. System 44 audiobooks #Ss @DO®@ 

» & |b Reading activity using System 44 materials O» 44 Book #Ss @O®@ 
s 3 O. Other practice activity using System 44 materials O. Decodable Digest #Ss @O®@ 
E ee SSG ETA Ee Ol, SAM Resources/Worksheets #Ss OD® 
= O Non-System 44 instructional activity Oh System 44 paperback books (without audio) | #Ss @O®@ 
= — |; Non-instructional activity 0; Letter manipulatives #Ss OO® 
O, Sound and articulation DVD # Ss OO®@ 


> Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: 


Interval end time: 


Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


What Is the teacher doing, and how fully is the teacher 
implementing each activity? 


Teacher-Led Support 


What supports is the teacher using, 
and how tully” is the teacher using them? 


0), Teaching Guide* O1, Decodable Digest O©O®@ 

e Introducing @O®@ | 0. 44 Book @DO®@ 

aol e Teaching/modeling @@®@ | Oa Teaching Resources for the System 44 Library @D®@ 

> 5 e Engaging students in guided practice | ©O@ | O, Letter manipulatives @Oe@ 

os e Apply and reinforcing/reviewing @O®@ | O; Flip chart @©O® 

CO | Notes. O, Sound and articulation DVD @©O®@ 

i) = * The “I Do, We Do, You Do” components should be incorporated for full 

E 2 Neeeetne Oj, SAM Resources/Worksheets ODO® 

E ? Full use of supports involves modeling answers and guiding student 0; System ae paperback books @©O®@ 

~*= usage of lessons; monitoring student work; and providing verbal or written OO, SAM Data: Reading Progress Report OO®@ 
feedback. If a teacher assigns an activity using a material but does not ! = = : = 

provide any guidance, monitoring, or feedback, mark 0. O;, SAM Data: Differentiated Instruction Grouping Report ODO 

Om Non-System 44 materials 0) SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


System 44 computer software ( students) 


© Not at all engaged/none engaged 


0) Partially engaged/some engaged 


@ Fully engaged/all engaged 


Independent (Individual/Paired) Activity 


What are individual students or pairs 
doing during small group instruction? Mark all that apply. 


Independent (Individual/Paired) Student Support 


What supports are the students using, 


and how engaged? are students in their use of them? 


O. Writing activity using System 44 materials O. System 44 audiobooks #Ss @DO®@ 

» & |b Reading activity using System 44 materials O» 44 Book #Ss @O®@ 
s 3 O. Other practice activity using System 44 materials O. Decodable Digest #Ss @O®@ 
E ee SSG ETA Ee Ol, SAM Resources/Worksheets #Ss OD® 
= O Non-System 44 instructional activity Oh System 44 paperback books (without audio) | #Ss @O®@ 
= — |; Non-instructional activity 0; Letter manipulatives #Ss OO® 
O, Sound and articulation DVD # Ss OO®@ 


> Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: Interval end time: Total minutes: 


System 44 Classroom Visit Protocol 


Teacher-Led Activity 


What Is the teacher doing, and how fully is the teacher 
implementing each activity? 


Teacher-Led Support 


What supports is the teacher using, 
and how tully? is the teacher using them? 
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O, Teaching Guide* O1, Decodable Digest O©O®@ 
e Introducing @O®@ | 0. 44 Book @D®@ 
e Teaching/modeling @@@ | Oy Teaching Resources for the System 44 Library @O®@ 
e Engaging students in guided practice | ©O®@ | Oe Letter manipulatives ©O@ 
e Apply and reinforcing/reviewing @@O®@ | 0; Flip chart @OD®@ 

Notes: O, Sound and articulation DVD @O@O®@ 

* The “I Do, We Do, You Do” components should be incorporated for full ©. SAM Resources/Worksheets ODO®@ 

implementation. i 

? Eull use of supports involves modeling answers and guiding student Oi System cms paperback books @Q@O®@ 

usage of lessons; monitoring student work; and providing verbal or written O, SAM Data: Reading Progress Report OO® 

feedback. If a teacher assigns an activity using a material but does not : = = = = 

provide any guidance, monitoring, or feedback, mark 0. O, SAM Data: Differentiated Instruction Grouping Report ODO 

Om Non-System 44 materials 01 SAM Data: Response to Intervention Report @O®@ 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


System 44 computer software 


How engaged are students in their use of the software? 


Individual/Pairs 


students) 


( 


System 44 computer software ( students) © Notat all engaged/none engaged ® Partially engaged/some engaged @ Fully engaged/all engaged 
Independent (Individual/Paired) Activity Independent (Individual/Paired) Student Support 
What are individual students or pairs What supports are the students using, 

doing during small group instruction? Mark all that apply. and how engaged? are students in their use of them? 
Of Writing activity using System 44 materials O System 44 audiobooks #Ss_ |: ODO 
O») Reading activity using System 44 materials O» 44 Book #Ss_ | ODO 
O. Other practice activity using System 44 materials O. Decodable Digest #Ss_ | ODO 
Oq Self monitoring using progress check document, O., SAM Resources/Worksheets #Ss OdO@ 

reading logs, Reading Counts quizzes, etc. a 
O Non-System 44 instructional activity O. System 44 paperback books (without audio) |#Ss__ ss |: OD 
O; Non-instructional activity O; Letter manipulatives #Ss_ | OM 
O, Sound and articulation DVD #Ss_ | ODO 


* Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


Interval start time: Interval end time: Total minutes: System 44 Classroom Visit Protocol 


Overall 
Did the teacher: Notes 

[1] Maintain an appropriate pace? @©O@® 
[2] Deliver lessons appropriate for the skill levels of the students? @©OO@® 
[3] Appear well prepared for the lesson? @©OO@® 
[4] Actively tries to engage students throughout the lesson? @©O@® 
[5] Maintain a positive learning environment (e.g., limited interruption, ODO® 
good classroom management)? 

[6] Execute the lesson well? @©O@® 


[7] Monitor students? (Check for on-task behavior, understanding; provide 
feedback) @0O® 


OVERVIEW 


[8] Keep students on task? @©O@® 


[9] Ensure smooth and efficient transitions between rotations (e.g., use of 
timer or clock, evidence of transition routines, rituals for students putting OO@® 
materials away). 


[10] Use flexible groupings for students (using SAM data, anecdotal ODO® 
evidence) or regroup students based on instructional needs? 


[11 Differentiate support based on students’ needs (e.g., provided @DO® 
individualized instruction, used ELL or SPED sections of teaching guide) 


4-point scale (0 = Not at all, 1 = To a small extent, 2 = To a moderate extent, 3 = Definitely) 


System 44 Classroom Setup Notes 
Computers (for 1/3 of class) are accessible and functioning @O®@ 
” 
g - Auxiliary equipment (headsets, microphones, CD players) are accessible ODO 
_i O | and functioning 
Oa 
System 44 materials are easily accessible to students and teacher @O®@ 


3-point scale (0 = None, 1= Some, 2 = All) 


Interval start time: Interval end time: Total minutes: System 44 Classroom Visit Protocol 


OO Check here if no whole class activity takes place on date of observation O Check here if whole class activity takes place at end of class 
Teacher-Led Activity Teacher-Led Support 
What is the teacher doing, and how fully is the What supports is the teacher using, 
teacher implementing each activity? and how fully” is the teacher using them? 

Of, Teaching Guide” O}, Letter manipulatives O©DO® 
= e Introducing @DO®@ O)/. Flip chart O@D®@ 
62 e Teaching/modeling @©O@ | Oa 44 Book O©O®@ 
go e Engaging students in guided practice @O®@ O). Decodable Digest @O®@ 
s z e Apply and reinforcing/reviewing @O@DO® O; Teaching Resources for the System 44 Library @O@O®@ 
o NoleS. O»1,, System 44 paperback books @O®@ 
5 ; The “I Do, We Do, You Do” components should be incorporated for full On SAM Resources/Worksheets @QO®@ 
o esha O; Sound and articulation DVD O©D®@ 
fo) Full use of supports involves modeling answers and guiding student usage of ; 

s lessons; monitoring student work; and providing verbal or written feedback. If a Oj SAM Data Reports (list_ i) @©O®@ 
S teacher assigns an activity using a material but does not provide any guidance, : 
monitoring, or feedback, mark 0. Ox Non-System 44 materials OD® 


3-point scale (0 = None, 1 = Partially, 2 = Fully) 


Independent (Individual/Paired) Activity Independent (Individual/Paired) Student Support 
What are individual students or pairs What supports are the students using, 

doing during large group instruction? Mark all that apply. and how engaged? are students in their use of them? 
Of Writing activity using System 44 materials O. System 44 audiobooks #Ss @DO®@ 
» & |b Reading activity using System 44 materials O» 44 Book #Ss @O®@ 
i 3S O. Other practice activity using System 44 materials O. Decodable Digest #Ss @O®@ 
s | Ou Sana er a ee nee cn ella O. SAM Resources/Worksheets #Ss O©D@ 
= O. Non-System 44 instructional activity Oh System 44 paperback books (without audio) | #Ss @O®@ 
= _, |G; Non-instructional activity O; Letter manipulatives #Ss ODO®@ 
O, Sound and articulation DVD # Ss @O@O® 


3 Student engagement: (0 = Not at all engaged/none engaged, 1 = Partially engaged/some engaged, 2 = Fully engaged/all engaged). 


School: 


System 44--Saginaw 
Teacher Interview 
Spring 2012 


Thank you for taking the time to talk with me today. As part of the evaluation of the 
System 44 program in your district, RMC Research staff are talking with teachers who 
implemented the program to help us understand factors that may have contributed to or 
hindered the success of the program. Your responses will be kept confidential and will 
not be shared with your principal or other district personnel. Your answers will be 
combined with other teachers’ information so that no one can know about any one 
person’s responses. 


Do you have any questions before we begin? 


1. Had you taught System 44 before this school year? 


Oj yes Ono 


If yes, how many years have you used the program? 


2. What features of the System 44 program do you like best? 


3. What aspects of System 44 implementation do you think you struggled with the 
most this year? How did you address or resolve the challenge(s)? 


4. How did you decide how to group students on any given day (e.g., which 
students would use the computer first)? 


RMC Research Corporation@Portland, OR 


How frequently did you use whole group instruction? (If once a week or more: 
what did you typically focus on during whole group instruction?) (If less than once 
a week: why did you not use whole group instruction more frequently?) 


In what ways, if any, did you use SAM reports? Which reports did you use, and 
how frequently did you use them? Prompt: (Show report examples) 


a. Screening and Placement; 

b. Software Performance; 

c. Reading Progress; 

d. Response to Intervention; 

e. Differentiated Instruction Grouping; 


f. Student Mastery 


What technology issues did you experience beyond the start-up period? Did 
these issues get in the way of implementing the program? 


Did you experience any issues with scheduling students to participate in 
System 44? If so, please explain whether these issues resolved during the 
school year. 


If you had the opportunity to make recommendations to teachers in another 
school that is planning to implement System 44 for the first time next year, what 
advice would you give them? 


System 44 Teacher Interview 


School: 


System 44—Saginaw 
Principal Interview 
Spring 2012 


As part of the evaluation of the Scholastic System 44 program, we want to talk with 
administrators at each participating school to learn more about the context in which System 44 
was implemented as well as factors that facilitated or hindered implementation of the program. 


1. In what year did your school first implement the System 44 reading program? 


OF 2011-2012 
i) 


2. In what year did your school first implement the Scholastic Read 180 reading program? 


OF 2011-2012 
0 


3. Have you had an opportunity to observe System 44 classes at your school? If yes, what 
features of the program do you think contribute to student success? 


A. What other programs or services are available to students in your school who are 
struggling with reading? (Prompts: in-class programs, pull-out programs, after school 
programs or tutoring) 


5. What factors do you think have facilitated implementation of System 44? (Prompts: 
district staff support, Scholastic staff support, teacher interest, student interest, 
computer-delivered instruction, teacher-delivered instruction, books and other materials) 
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10. 


11. 


What factors have hindered implementation of System 44? (Prompts: hardware issues, 
software issues, problems with other materials such as books, testing to identify the right 
students, testing for treatment and control group students, pullout from other subjects, 
research requirement for random assignment, problems at Thompson and Curran) 


What feedback have you received from teachers or parents about student outcomes 
resulting from participation in System 44?) 


What feedback have you received from teachers about the training or support they 
received from Scholastic for implementing the program? 


What plans does your school have for continuing System 44 next year? (Prompts: Do 
you plan to do anything differently as far as implementation or scheduling?) 


Do you have any suggestions for how the evaluation might have been better 
implemented? (e.g., identifying students, random assignment, SPI and SRI testing, 
individual testing of treatment and control group students). 


Are there any other comments you would like to make? 


System 44 Principal Interview 


System 44—Saginaw 
District Staff Focus Group 
Spring 2012 


As part of the evaluation of the Scholastic System 44 program, we want to talk with 
district staff involved in implementing the program to learn more about the context in 


which System 44 was implemented as well as factors that facilitated or hindered 
implementation of the program. 


System 44 Implementation 


1. What types of professional development did System 44 teachers receive this year? 


What did you or the System 44 teachers find most useful about the professional 
development they received? 


What suggestions, if any, do you have regarding professional development? 


2. What factors do you think facilitated implementation of System 44 in the 16 study 
schools this year? 


3. What factors do you think hindered implementation (i.e., what challenges did schools 
face in implementing System 44 in their classrooms)? 


4. What recommendations would you have for another district that is starting to use the 
System 44 program in their schools (i.e., what were the lessons learned)? 


5. Overall, what impact do you think the program has had on students? What aspects of 
the program do you believe have had the greatest impact on students? 


Evaluation Implementation 


6. What questions would you be interested in having answered by the evaluation of System 
44 in your district? 


7. What recommendations would you make concerning the evaluation component of the 
System 44 study (i.e., what were the lessons learned)? 
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8. What was the process for administering the SPI and SRI tests in Fall and Winter (i.e., 
who was responsible for administering them, how long did the process take)? 


9. Is there anything else that you would like to add? 


RMC Research Corporation ® Portland, OR 


Appendix C 
Item Results for Instructional Practices 


Percentage of Teachers Using Various Strategies With Struggling Readers 


Exhibit C1 


System 44 System 44 Control Teachers 
Last Year (2010-2011) | This Year (2011-2012) This Year (2011-2012) _ 
: Not Small Not Small Not Small 
Instructional Strategy Part Part Central Part Part Central Part Part Central 
Instruction 1. 2. 4. 5. 7. 8. 
1. Provide time in reading block for students 0 59 41 0 15 85 0 38 62 
to practice skills on their own. 
2. Provide materials for at-home practice of 18 65 18 68 21 11 18 50 32 
skills introduced in class. 
3. Provide extra reading instructional time 0 24 76 10 30 60 3 31 66 
for struggling readers. 
4. Include writing opportunities in reading 0 47 53 10 50 40 0 28 72 
instruction. 
5. Build spelling practice into reading 12 53 35 15 50 35 14 41 45 
instruction. 
6. Develop reading skills using science and 18 47 35 65 10 25 3 34 62 
social studies texts. 
Grouping 
7. Teach whole-class reading lessons. 24 35 41 0 25 75 7 29 64 
8. Work one-to-one with students on 0 41 59 25 75 4 68 29 
reading. 
9. Work with small groups of students. 0 18 82 0 15 85 0 41 59 
10. Group students based on skill levels. 12 29 59 15 85 11 43 46 
11. Group students based on need for 6 29 65 25 70 0 55 45 
additional instruction in specific, targeted 
skills. 
12. Group students based on mixed abilities 24 41 35 15 30 55 0 48 52 
(pairs or cooperative groups). 
Reading Materials 
13. Use core reading series. 0 41 59 20 5 75 10 45 45 
14. Use supplementary reading materials. 0 18 82 30 20 50 0 28 72 
15. Use trade books. 24 53 24 55 20 25 7 46 46 
16. Use books that are easy to decode. 6 35 59 25 30 45 11 57 32 
17. Use separate intervention materials for 0 24 76 45 20 35 3 41 55 
some students. 
18. Use reading software/technology. 35 35 29 15 15 70 14 41 45 
19. Use teacher-made materials. 6 59 35 55 30 15 7 52 41 
Assessments 
20. Use test results to organize instructional 12 35 53 11 21 68 3 17 79 
groups. 
21. Use informal reading inventories. 12 41 47 47 26 26 14 45 41 
22. Conduct miscue analysis, analyzing 6 29 65 11 42 47 45 52 
errors students make while reading 
aloud. 
23. Use tests to determine progress on skills. 0 24 76 0 47 53 0 48 52 
24. Use diagnostic tests to identify students 6 12 82 50 44 3 28 69 
who need reading intervention services. 
RMC Research Corporation @Portland, OR 93 


Exhibit C2 
Percentage of Teachers Using Various Activities With Struggling Readers 


System 44 System 44 is) | 
Last Year (2010-2011) (This Year (2011-2012) 12) 
; oe Not Small Not Small 

Instructional Activity Part Part Central Part Part Central Part Part Central 

Reading Text 10. 11. 12. 13. 14. 15. 16. 17. 18. 

25. Students reread familiar text. 0 24 76 0 20 80 3 28 69 

26. Students confirm or revise predictions 0 35 65 5 45 50 3 41 55 
after reading. 

27. Students generate their own questions 18 53 29 20 40 40 7 45 48 
about text material. 

28. Students identify their comprehension 29 47 24 35 45 20 31 31 38 
breakdowns and use fix-up strategies 
with a partner. 

29. Students orally summarize main events in 0 29 71 5 30 65 7 34 59 
stories and informational texts. 

30. Students use graphic and semantic 12 35 53 20 40 40 3 31 66 
organizers to track information. 

Work With Sounds and Words 

31. | teach specific strategies for decoding 6 29 65 0 10 90 10 52 38 
unfamiliar words. 

32. | teach decoding/phonics skills while 6 24 71 0 5 95 17 41 41 
reading stories. 

33. Students practice reading high frequency 6 24 71 0 25 75 17 48 34 
words for automaticity. 

34. Students use knowledge of root words, 0 41 59 0 15 85 0 45 55 
prefixes, and suffixes to decode new 
words. 

35. Students work with prefixes and suffixes 0 53 47 0 35 65 3 62 34 
to change the meaning of words. 

36. Students use context clues to identify 0 18 82 0 20 80 0 31 69 
unknown words. 

37. | discuss new and unusual words before 6 29 65 0 16 84 0 34 66 
reading. 

Other Techniques 

38. Students answer questions in writing after 12 35 53 10 40 50 0 24 76 
reading stories. 

39. Students select books from the library for 0 24 76 11 21 68 3 21 76 
independent reading. 

40. Students are given time to read on their 0 35 65 15 15 70 3 17 79 


own for enjoyment. 
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Appendix D 
Fixed and Random Effects and Intraclass Correlations 


Exhibit D1 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOSREC Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 18.29 0.89 15 20.64 .000 

Student System 44 0.05 113 308 0.05 .965 
Fall TOSREC Score 0.64 0.05 308 12.97 .000 
Learning disabled -3.09 1.29 308 -2.40 .017 
African American 0.08 1.62 308 0.05 .960 
English language learner -3.05 2.97 308 -1.03 805 
Elementary school level 3.46 1.64 308 2.11 .035 

Random Effects 

Level Variance Components Variance 

School Level 2 5.11 

Student Level 1 98.80 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 6.86 .037 

Student Level 1 178.77 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D2 


Final Model for Estimating Fixed and Random Effects of System 44 
on Spring CTOPP Elision Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 18.76 0.36 15 51.84 000 

Student System 44 1.45 0.45 310 3.25 002 
Fall CTOPP Elision Score 0.69 0.04 310 16.32 000 
Learning disabled -0.87 0.47 310 -1.86 .064 
African American -1.47 0.64 310 -2.28 023 
English language learner -2.77 1.18 310 -2.34 020 
Elementary school level -1.54 0.65 310 -2.36 019 

Random Effects 

Level Variance Components Variance 

School Level 2 0.90 

Student Level 1 15.60 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 1.80 .053 

Student Level 1 32.10 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D3 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOWRE Sight Word Efficiency Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 57.85 0.40 15 145.16 .000 

Student System 44 2.03 0.69 309 2.93 .004 
Fall TOWRE Sight Word 
Efficiency Score 0.85 0.03 309 30.37 .000 
Learning disabled -1.91 0.78 309 -2.44 015 
African American 0.13 0.94 309 0.14 .893 
English language learner -1.00 1.80 309 -0.56 579 
Elementary school level 1.56 0.83 309 1.89 .060 

Random Effects 

Level Variance Components Variance 

School Level 2 0.38 

Student Level 1 37.50 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 10.35 .052 

Student Level 1 186.99 


*The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D4 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring TOWRE Phonetic Decoding Efficiency Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 23.98 0.82 15 29.27 000 

Student System 44 1.27 0.68 308 1.85 .065 
Fall TOWRE Phonetic 
Decoding Efficiency Score 0.85 0.03 308 26.14 000 
Learning disabled -2.08 0.74 308 -2.80 006 
African American -0.15 1.01 308 -0.15 .881 
English language learner -0.51 1.82 308 -0.28 780 
Elementary school level -2.08 0.74 308 -2.80 006 

Random Effects 

Level Variance Components Variance 

School Level 2 7.10 

Student Level 1 36.05 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 7.42 .053 

Student Level 1 132.84 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D5 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SRI Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 404.78 18.23 15 22.20 .000 

Student System 44 80.82 18.73 280 4.31 .000 
Fall SRI Score 0.69 0.04 280 16.44 .000 
Learning disabled -57.20 20.69 280 -2.77 .007 
African American -32.45 26.53 280 -1.22 .223 
English language learner -58.14 48.58 280 -1.20 .233 
Elementary school level -58.40 32.04 280 -1.82 .069 

Random Effects 

Level Variance Components Variance 

School Level 2 2,829.18 

Student Level 1 24,318.87 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 8,930.27 .125 

Student Level 1 62,548.08 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D6 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Letter Name Accuracy Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 10.74 0.03 15 357.05 .000 

Student System 44 0.03 0.06 272 0.45 .653 
Fall SPI Letter Name 
Accuracy Score 0.10 0.03 272 3.09 .003 
Learning disabled -0.01 0.06 272 -0.15 .879 
African American -0.04 0.08 272 -0.56 575 
English language learner 0.09 0.17 272 0.57 571 
Elementary school level -0.06 0.06 272 -0.92 357 

Random Effects 

Level Variance Components Variance 

School Level 2 0.00 

Student Level 1 0.25 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 0.00 .000 
Student Level 1 0.26 


*The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D7 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Sight Word Accuracy Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 22.51 0.31 15 71.58 .000 

Student System 44 -0.25 0.39 272 -0.64 525 
Fall SPI Sight Word 
Accuracy Score 0.76 0.05 272 15.50 .000 
Learning disabled -0.92 0.45 272 -2.07 .039 
African American 0.86 0.54 272 1.58 115 
English language learner -1.19 1.05 272 -1.13 .258 
Elementary school level -0.24 0.58 272 -0.42 .677 

Random Effects 

Level Variance Components Variance 

School Level 2 0.73 

Student Level 1 10.13 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 3.48 .128 

Student Level 1 23.68 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D8 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Sight Word Fluency Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 8.59 0.37 15 23.43 000 

Student System 44 0.85 0.46 272 1.85 .065 
Fall SPI Sight Word 
Fluency Score 0.71 0.07 272 10.34 000 
Learning disabled -2.18 0.50 272 -4.36 000 
African American 1.12 0.64 272 1.73 .084 
English language learner 2.27 1.25 272 1.82 .070 
Elementary school level -0.59 0.68 272 -0.86 391 

Random Effects 

Level Variance Components Variance 

School Level 2 0.96 

Student Level 1 14.37 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 0.69 .027 

Student Level 1 24.67 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D9 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Nonsense Word Accuracy Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 20.27 0.23 15 86.35 .000 

Student System 44 1.61 0.43 272 3.74 .000 
Fall SP! Nonsense Word 
Accuracy Score 0.63 0.05 272 11.92 .000 
Learning disabled -1.36 0.46 272 -2.98 .004 
African American -1.68 0.58 272 -2.92 .004 
English language learner -1.01 1.16 272 -0.87 .387 
Elementary school level -1.14 0.48 272 -2.37 .018 

Random Effects 

Level Variance Components Variance 

School Level 2 0.09 

Student Level 1 12.84 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 2.20 .089 

Student Level 1 22.49 


*The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D10 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Nonsense Word Fluency Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 6.81 0.47 15 14.45 .000 

Student System 44 0.42 0.47 272 0.90 372 
Fall SPI Nonsense Word 
Fluency Score 0.55 0.08 272 6.99 .000 
Learning disabled -1.72 0.49 272 -3.52 .001 
African American -0.77 0.68 272 -1.15 .253 
English language learner 2.44 1.30 272 1.88 .060 
Elementary school level -1.73 0.77 272 -2.26 025 

Random Effects 

Level Variance Components Variance 

School Level 2 2.12 

Student Level 1 15.04 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 1.02 .049 

Student Level 1 19.79 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 


term on the right-hand side of the model. 
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Exhibit D11 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Overall Accuracy Scores 


Fixed Effects 


Level Effect Impact (B) SE DF t p 

School Intercept 42.74 0.51 15 83.77 .000 

Student System 44 1.43 0.67 272 2.12 .035 
Fall SPI Overall Accuracy 
Score 0.79 0.05 272 15.56 .000 
Learning disabled -1.63 0.78 272 -2.09 .037 
African American -0.69 0.94 272 -0.73 465 
English language learner -1.86 1.83 272 -1.02 310 
Elementary school level -0.80 0.97 272 -0.82 412 

Random Effects 

Level Variance Components Variance 

School Level 2 1.70 

Student Level 1 30.97 

Random Effects (From Unconditional Model)* 

Level Variance Components Variance ICC 

School Level 2 11.27 138 

Student Level 1 70.36 


“The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Exhibit D12 
Final Model for Estimating Fixed and Random Effects of System 44 
on Spring SPI Overall Fluency Scores 


Fixed Effects 


Level Effect Impact (f) SE DF t p 

School Intercept 15.54 0.79 15 19.57 .000 

Student System 44 1.35 0.80 272 1.69 091 
Fall SPI Overall Fluency 
Score 0.76 0.08 272 9.84 .000 
Learning disabled -3.60 0.86 272 -4.19 .000 
African American 0.52 1.14 272 0.46 .649 
English language learner 5.20 2.18 272 2.39 .018 
Elementary school level -2.26 1.31 272 -1.72 .085 

Random Effects 

Level Variance Components Variance 

School Level 2 6.01 

Student Level 1 42.78 


Random Effects (From Unconditional Model)* 


Level Variance Components Variance Icc 
School Level 2 2.55 .035 
Student Level 1 70.34 


*The unconditional model is a 2-level model with students (Level 1) nested in schools (Level 2), and only an intercept 
term on the right-hand side of the model. 
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Appendix E 
Mean Scores on Reading Tests, SRI, and SPI 
Overall Accuracy and Fluency Outcomes 


Exhibit E1 
TOSREC Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 12.02 12.85 0.83 28 13.09 15.07 1.98 
5 54 13.94 19.47 5.53 59 15.11 22.98 7.87 
6 33 22.34 20.49 -1.85 35 23.23 19.45 -3.77 
7 19 12.79 22.35 9.56 17 17.26 23.81 6.55 
8 18 10.83 13.04 2.21 23 9.84 8.01 -1.83 
Decoding Status 
Pre decoder 2 14.10 13.65 -0.45 5 15.42 17.10 1.68 
Beginning decoder 71 12.11 15.02 2.91 63 9.29 12.66 3.38 
Developing decoder 82 17.23 20.61 3.38 94 20.51 23.02 2.51 
FRL Status 
None 7 13.27 15.61 2.34 6 26.35 28.20 1.85 
Free or Reduced-Price 148 14.94 18.05 3.11 156 15.59 18.45 2.86 
ELL Status 
English only 149 14.89 18.19 3.30 153 15.86 18.91 3.05 
English learner 6 14.15 11.67 -2.48 9 18.19 17.08 -1.11 
SPED Status 
None 71 22.28 24.25 1.97 81 20.80 23.93 3.13 
Learning disability 84 8.51 12.55 4.03 81 11.18 13.69 2.51 
Ethnicity 
Caucasian 16 19.87 21.80 1.93 17 18.48 19.59 1.11 
African American 124 13.71 17.36 3.65 124 15.11 18.46 3.35 
Hispanic 15 18.97 18.87 -0.10 21 19.16 20.22 1.07 
School 
Coulter 5 8.78 7.00 -1.78 5 16.26 19.10 2.84 
Heavenrich 4 11.60 11.65 0.05 6 19.53 15.95 -3.58 
Herig 12 22.76 23.60 0.84 15 18.63 20.57 1.93 
Houghton 5 8.94 15.94 7.00 7 5.83 17.31 11.49 
Jerome 7 7.18 10.04 2.86 6 16.02 16.55 0.53 
Kempton 5 9.34 21.32 11.98 7 16.03 27.04 11.01 
Longfellow 5 8.28 13.22 4.94 6 15.47 25.75 10.28 
Loomis 18 17.07 22.94 5.87 18 21.11 24.65 3.54 
Merrill Park 6 9.98 13.07 3.08 3 9.43 5.80 -3.63 


exhibit continues 


RMC Research Corporation @Portland, OR 111 


Exhibit E1 (continued) 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 16.93 27.00 10.08 3 15.67 24.30 8.63 
Rouse 8 16.63 12.88 -3.75 7 11.80 16.07 4.27 
Stone 4 3.35 18.88 15.53 3 1.00 11.77 10.77 
Arthur Eddy 11 18.04 20.20 2.16 9 12.13 17.51 5.38 
Zilwaukee 3 21.47 18.07 -3.40 8 21.36 27.24 5.88 
Ruben Daniels 37 15.38 19.19 3.81 37 12.87 15.20 2.34 
Thompson 21 16.00 15.54 -0.46 22 21.52 16.52 -5.00 
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Exhibit E2 
CTOPP Elision Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 15.71 16.32 0.61 28 16.82 16.46 -0.36 
5 54 19.15 19.59 0.44 59 19.54 18.32 -1.22 
6 33 19.79 21.58 1.79 35 19.74 19.89 0.14 
7 19 15.58 18.11 2.53 17 17.71 18.24 0.53 
8 18 18.56 21.11 2.56 23 18.65 19.30 0.65 
Decoding Status 
Pre decoder 2 12.00 14.50 2.50 5 16.20 17.60 1.40 
Beginning decoder 71 17.21 17.69 0.48 63 17.87 16.98 -0.89 
Developing decoder 82 19.00 20.91 1.91 94 19.55 19.51 -0.04 
FRL Status 
None 7 16.71 19.14 2.43 6 20.67 19.83 -0.83 
Free or Reduced-Price 148 18.16 19.36 1.21 156 18.72 18.42 -0.31 
ELL Status 
English only 149 18.19 19.56 1.37 153 18.84 18.51 -0.33 
English learner 6 15.50 14.17 -1.33 9 18.00 17.78 -0.22 
SPED Status 
None 71 19.90 21.01 1.11 81 19.35 19.41 0.06 
Learning disability 84 16.56 17.95 1.39 81 18.25 17.53 -0.72 
Ethnicity 
Caucasian 16 21.19 22.69 1.50 17 20.53 19.53 -1.00 
African American 124 17.73 19.08 1.35 124 18.52 17.96 -0.56 
Hispanic 15 17.80 18.07 0.27 21 19.05 20.62 1.57 
School 
Coulter 5 22.80 17.20 -5.60 5 22.40 14.60 -7.80 
Heavenrich 4 19.25 18.25 -1.00 6 17.33 17.50 0.17 
Herig 12 19.50 20.17 0.67 15 16.93 18.73 1.80 
Houghton 5 17.20 18.80 1.60 7 20.43 18.14 -2.29 
Jerome 7 17.14 17.71 0.57 6 17.17 15.17 -2.00 
Kempton 5 18.40 18.20 -0.20 7 18.14 18.86 0.71 
Longfellow 5 21.60 21.40 -0.20 6 19.33 22.33 3.00 
Loomis 18 18.33 19.33 1.00 18 18.67 17.50 -1.17 
Merrill Park 6 14.67 14.83 0.17 3 12.67 11.33 -1.33 
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Exhibit E2 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 15.75 18.50 2.75 3 16.33 17.00 0.67 
Rouse 8 15.50 17.75 2.25 7 20.00 19.29 -0.71 
Stone 4 14.00 14.50 0.50 3 17.00 12.67 -4.33 
Arthur Eddy 11 17.36 19.73 2.36 9 21.11 18.11 -3.00 
Zilwaukee 3 22.00 22.67 0.67 8 23.88 22.38 -1.50 
Ruben Daniels 37 18.65 20.70 2.05 37 18.24 19.38 1.14 
Thompson 21 17.38 20.05 2.67 22 18.91 18.95 0.05 
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Exhibit E3 
TOWRE Sight Word Efficiency Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 40.33 48.44 8.10 28 44.13 51.43 7.30 
5 54 49.73 57.68 7.94 59 54.22 58.64 4.42 
6 33 57.36 62.32 4.95 35 57.09 58.77 1.69 
7 19 56.74 62.26 5.53 17 57.76 60.18 2.41 
8 18 56.06 59.64 3.58 23 56.91 61.70 4.78 
Decoding Status 
Pre decoder 2 40.00 44.50 4.50 5 45.80 54.40 8.60 
Beginning decoder 71 42.71 49.96 7.26 63 44.75 49.56 4.81 
Developing decoder 82 58.61 64.54 5.93 94 60.37 63.87 3.50 
FRL Status 
None 7 45.50 52.64 7.14 6 61.83 62.17 0.33 
Free or Reduced-Price 148 51.41 57.84 6.43 156 53.54 57.86 4.31 
ELL Status 
English only 149 51.32 57.79 6.46 153 53.73 58.00 4.27 
English learner 6 46.67 53.17 6.50 9 55.94 58.28 2.33 
SPED Status 
None 71 58.13 64.41 6.28 81 59.73 64.23 4.50 
Learning disability 84 45.16 51.86 6.69 81 47.96 51.80 3.83 
Ethnicity 
Caucasian 16 51.50 58.28 6.78 17 50.03 54.74 4.71 
African American 124 51.11 57.54 6.42 124 54.10 58.33 4.23 
Hispanic 15 50.97 57.47 6.50 21 55.48 58.83 3.36 
School 
Coulter 5 42.10 45.10 3.00 5 56.60 62.10 5.50 
Heavenrich 4 48.13 53.25 5.13 6 49.58 53.67 4.08 
Herig 12 54.71 61.88 7.17 15 53.37 60.70 7.33 
Houghton 5 39.90 47.30 7.40 7 54.00 56.43 2.43 
Jerome 7 41.07 47.21 6.14 6 48.25 55.33 7.08 
Kempton 5 49.40 56.90 7.50 7 55.21 60.43 5.21 
Longfellow 5 41.20 52.50 11.30 6 55.08 61.00 5.92 
Loomis 18 53.78 62.11 8.33 18 54.06 56.61 2.56 
Merrill Park 6 39.25 56.17 16.92 3 30.67 34.67 4.00 
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Exhibit E3 (continued) 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 52.25 59.00 6.75 3 61.00 66.33 5.33 
Rouse 8 42.69 48.88 6.19 7 43.43 49.86 6.43 
Stone 4 48.50 53.38 4.88 3 39.83 36.83 -3.00 
Arthur Eddy 11 53.41 58.86 5.45 9 55.11 60.22 5.11 
Zilwaukee 3 49.50 58.83 9.33 8 51.38 59.81 8.44 
Ruben Daniels 37 55.80 61.31 5.51 37 52.59 56.69 4.09 
Thompson 21 56.00 59.33 3.33 22 65.16 65.45 0.30 
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Exhibit E4 
TOWRE Phonetic Decoding Efficiency Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 15.66 18.94 3.27 28 17.80 20.59 2.79 
5 54 22.16 24.64 2.48 59 21.63 23.85 2.22 
6 33 22.41 26.55 4.14 35 24.21 26.13 1.92 
7 19 19.45 23.50 4.05 17 21.59 23.59 2.00 
8 18 20.72 24.19 3.47 23 22.78 26.04 3.26 
Decoding Status 
Pre decoder 2 6.00 12.25 6.25 5 18.60 23.70 5.10 
Beginning decoder 71 15.91 19.28 3.37 63 16.15 17.50 1.35 
Developing decoder 82 24.66 27.77 3.10 94 25.56 28.39 2.83 
FRL Status 
None 7 16.36 21.50 5.14 6 29.50 33.17 3.67 
Free or Reduced-Price 148 20.60 23.81 3.21 156 21.38 23.70 2.31 
ELL Status 
English only 149 20.57 23.94 3.37 153 21.50 23.84 2.34 
English learner 6 16.50 17.92 1.42 9 24.78 27.61 2.83 
SPED Status 
None 71 23.16 27.61 4.45 81 25.48 28.27 2.80 
Learning disability 84 18.09 20.46 2.37 81 17.90 19.78 1.88 
Ethnicity 
Caucasian 16 23.03 26.72 3.69 17 21.09 23.12 2.03 
African American 124 20.21 23.32 3.11 124 21.30 23.46 2.16 
Hispanic 15 19.33 23.70 4.37 21 24.45 28.26 3.81 
School 
Coulter 5 18.50 21.10 2.60 5 19.40 22.90 3.50 
Heavenrich 4 18.88 17.67 -1.21 6 19.58 18.75 -0.83 
Herig 12 20.71 26.42 5.71 15 22.50 28.60 6.10 
Houghton 5 15.40 16.60 1.20 7 22.29 22.57 0.29 
Jerome 7 18.71 17.07 -1.64 6 19.25 20.58 1.33 
Kempton 5 19.70 24.10 4.40 7 20.07 26.93 6.86 
Longfellow 5 17.70 25.60 7.90 6 23.25 28.33 5.08 
Loomis 18 21.25 25.08 3.83 18 19.44 20.86 1.42 
Merrill Park 6 24.67 22.33 -2.33 3 10.17 8.17 -2.00 
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Exhibit E4 (continued) 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 12.38 16.75 4.38 3 25.17 30.17 5.00 
Rouse 8 19.06 24.19 5.13 7 21.57 22.57 1.00 
Stone 4 19.50 16.50 -3.00 3 15.83 5.50 -10.33 
Arthur Eddy 11 25.86 27.32 1.45 9 23.44 24.72 1.28 
Zilwaukee 3 21.33 28.33 7.00 8 22.00 30.44 8.44 
Ruben Daniels 37 19.42 23.47 4.05 37 20.76 22.38 1.62 
Thompson 21 22.62 26.60 3.98 22 27.27 29.07 1.80 
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Exhibit E5 
SRI Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 141.74 214.97 73.23 28 128.68 185.96 57.29 
5 54 238.19 373.44 135.26 59 326.25 404.73 78.47 
6 33 500.97 551.39 50.42 35 480.20 439.11 -41.09 
7 19 343.58 477.42 133.84 17 381.76 378.35 -3.41 
8 18 374.83 587.89 213.06 23 426.48 426.74 0.26 
Decoding Status 
Pre decoder 2 250.00 253.50 3.50 5 278.00 226.80 -51.20 
Beginning decoder 71 205.92 327.44 121.52 63 228.87 260.89 32.02 
Developing decoder 82 389.55 499.07 109.52 94 427.12 458.85 31.73 
FRL Status 
None 7 228.43 419.14 190.71 6 547.17 569.67 22.50 
Free or Reduced-Price 148 307.19 417.20 110.01 156 337.66 367.21 29.54 
ELL Status 
English only 149 304.36 417.51 113.15 153 348.13 379.85 31.72 
English learner 6 285.67 411.67 126.00 9 299.33 287.22 -12.11 
SPED Status 
None 71 435.52 546.30 110.77 81 416.63 469.56 52.93 
Learning disability 84 192.15 308.24 116.08 81 274.21 279.85 5.64 
Ethnicity 
Caucasian 16 371.81 472.69 100.88 17 353.06 458.06 105.00 
African American 124 285.89 399.46 113.57 124 339.70 360.06 20.36 
Hispanic 15 377.60 505.53 3127.93 21 373.00 393.67 20.67 
School 
Coulter 5 51.20 215.80 164.60 5 348.40 290.20 -58.20 
Heavenrich 4 253.25 219.00 -34.25 6 260.00 129.83 -130.17 
Herig 12 269.33 566.33 297.00 15 281.07 435.73 154.67 
Houghton 5 204.40 230.60 26.20 7 197.29 241.86 44.57 
Jerome 7 107.71 130.29 22.57 6 205.50 275.83 70.33 
Kempton 5 192.00 535.60 343.60 7 300.43 352.71 52.29 
Longfellow 5 119.20 187.40 68.20 6 298.83 422.83 124.00 
Loomis 18 379.56 422.11 42.56 18 349.11 515.89 166.78 
Merrill Park 6 146.83 326.50 179.67 3 60.67 145.00 84.33 
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Exhibit E5 (continued) 


Treatment Control 

Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 

Miller 4 328.00 501.25 173.25 3 383.67 293.67 -90.00 
Rouse 8 200.13 279.13 79.00 7 296.86 259.00 -37.86 
Stone 4 206.75 257.50 50.75 3 201.00 218.00 17.00 
Arthur Eddy 11 370.27 451.36 81.09 9 403.00 482.11 79.11 
Zilwaukee 3 404.33 419.67 15.33 8 483.25 505.25 22.00 
Ruben Daniels 37 400.49 482.65 82.16 37 400.14 390.54 -9.59 


Thompson 21 365.38 540.05 174.67 22 424.27 349.18 -75.09 
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Exhibit E6 
SPI Overall Accuracy Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 30.16 37.61 7.45 28 31.93 37.96 6.04 
5 54 34.94 41.49 6.55 59 35.88 43.17 7.29 
6 33 38.70 47.09 8.40 35 38.80 45.39 6.59 
7 19 36.05 44.83 8.78 17 37.00 42.67 5.67 
8 18 38.72 47.61 8.89 23 38.48 43.20 4.72 
Decoding Status 
Pre decoder 2 23.00 33.00 10.00 5 32.40 44.20 11.80 
Beginning decoder 71 31.87 39.13 7.26 63 31.95 38.76 6.81 
Developing decoder 82 38.68 46.67 7.99 94 39.45 45.03 5.59 
FRL Status 
None 7 32.00 40.57 8.57 6 43.33 50.33 7.00 
Free or Reduced-Price 148 35.52 43.14 7.62 156 36.04 42.35 6.30 
ELL Status 
English only 149 35.51 43.22 7.71 153 36.26 42.75 6.49 
English learner 6 31.67 38.17 6.50 9 37.22 41.22 4.00 
SPED Status 
None 71 39.55 47.23 7.68 81 39.60 45.80 6.20 
Learning disability 84 31.82 39.24 7.42 81 33.02 39.22 6.19 
Ethnicity 
Caucasian 16 36.25 45.47 9.22 17 35.35 41.47 6.12 
African American 124 35.34 42.83 7.49 124 36.37 42.75 6.37 
Hispanic 15 34.60 42.13 7.53 21 36.76 43.15 6.39 
School 
Coulter 5 27.20 38.20 11.00 5 37.40 46.80 9.40 
Heavenrich 4 31.50 42.25 10.75 6 37.50 39.00 1.50 
Herig 12 36.75 43.45 6.70 15 35.13 42.07 6.93 
Houghton 5 36.20 38.80 2.60 7 36.57 40.57 4.00 
Jerome 7 30.43 29.00 -1.43 6 31.50 40.83 9.33 
Kempton 5 34.40 47.80 13.40 7 35.71 44.43 8.71 
Longfellow 5 29.40 41.00 11.60 6 35.00 42.67 7.67 
Loomis 18 37.72 42.89 5.17 18 36.22 43.17 6.94 
Merrill Park 6 30.00 36.83 6.83 3 21.67 25.67 4.00 
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Exhibit E6 (continued) 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 36.25 45.00 8.75 3 41.33 47.33 6.00 
Rouse 8 33.25 42.38 9.13 7 32.29 40.29 8.00 
Stone 4 33.50 36.25 2.75 3 22.00 31.00 9.00 
Arthur Eddy 11 35.55 42.25 6.70 9 40.33 46.56 6.22 
Zilwaukee 3 33.67 44.33 10.67 8 39.63 45.13 5.50 
Ruben Daniels 37 37.70 47.14 9.43 37 36.59 43.55 6.95 
Thompson 21 36.86 45.10 8.24 22 39.64 43.62 3.98 
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Exhibit E7 
SPI Overall Fluency Means by Subgroup 


Treatment Control 
Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Grade 
4 31 7.77 11.39 3.61 28 8.86 10.71 1.86 
5 54 9.63 14.75 5.12 59 10.54 14.47 3.93 
6 33 13.55 16.41 2.86 35 13.31 15.12 1.81 
7 19 14.79 17.67 2.88 17 14.12 16.13 2.02 
8 18 14.50 14.83 0.33 23 15.22 18.00 2.78 
Decoding Status 
Pre decoder 2 5.50 8.00 2.50 5 6.60 15.00 8.40 
Beginning decoder 71 5.75 10.71 4.96 63 6.10 9.90 3.80 
Developing decoder 82 16.23 18.48 2.25 94 16.05 17.46 1.40 
FRL Status 
None 7 10.71 12.86 2.14 6 16.00 21.67 5.67 
Free or Reduced-Price 148 11.32 14.86 3.54 156 11.73 14.26 2.53 
ELL Status 
English only 149 11.39 14.58 3.19 153 11.93 14.45 2.52 
English learner 6 8.83 19.33 10.50 9 11.22 16.22 5.00 
SPED Status 
None 71 13.48 18.08 4.61 81 13.57 17.22 3.65 
Learning disability 84 9.44 11.78 2.34 81 10.21 11.62 1.41 
Ethnicity 
Caucasian 16 9.13 16.00 6.88 17 10.00 13.65 3.65 
African American 124 11.70 14.45 2.75 124 11.95 14.33 2.38 
Hispanic 15 10.20 16.07 5.87 21 13.05 16.60 3.55 
School 
Coulter 5 8.00 11.60 3.60 5 9.80 14.00 4.20 
Heavenrich 4 8.00 14.25 6.25 6 9.50 13.50 4.00 
Herig 12 9.75 18.55 8.80 15 8.00 12.20 4.20 
Houghton 5 2.60 7.20 4.60 7 6.86 9.57 2.71 
Jerome 7 6.57 10.14 3.57 6 8.83 13.00 4.17 
Kempton 5 8.00 15.60 7.60 7 14.14 18.00 3.86 
Longfellow 5 11.20 12.40 1.20 6 12.00 15.00 3.00 
Loomis 18 12.28 14.39 2.11 18 13.61 14.56 0.94 
Merrill Park 6 7.67 11.67 4.00 3 7.67 7.00 -0.67 
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Exhibit E7 (continued) 


Treatment Control 


Subgroup n Pretest Posttest Gain n Pretest Postiest Gain 
Miller 4 11.50 12.00 0.50 3 11.33 19.67 8.33 
Rouse 8 10.00 17.50 7.50 7 8.00 14.71 6.71 
Stone 4 6.00 12.25 6.25 3 5.00 5.33 0.33 
Arthur Eddy 11 12.00 23.00 11.00 9 12.89 13.33 0.44 
Zilwaukee 3 12.33 14.67 2.33 8 12.38 21.00 8.63 
Ruben Daniels 37 14.24 13.59 -0.65 37 12.76 13.45 0.69 
Thompson 21 13.95 17.60 3.65 22 16.73 18.76 2.03 
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Appendix F 
Effect Sizes for Tests By Reading Domain 


Exhibit F1 
Effect Sizes for Individual, SRI, and SPI Tests By Reading Domain 


Sample 
Learning Elementary Middle 

Reading Domain Overall Disabled School School 
Phonemic awareness 

CTOPP Elision 0.27** 0.36** 0.26* 0.30** 
Letter identification 

SPI Letter Name Accuracy 0.06 0.15 0.18 -0.21 
Phonics 

TOWRE Phonetic Decoding -0.09 -0.01 0.09 0.19 

Efficiency” 

TOWRE Sight Word Efficiency 0.16** 0.24* 0.177 0.24** 

SPI Sight Word Accuracy -0.05 -0.02 -0.16 0.12 

SPI Sight Word Fluency 0.17° 0.28* 0.18 0.16 

SPI Nonsense Word Accuracy 0.32*** 0.36** 0.10 0.59*** 

SPI Nonsense Word Fluency 0.10 0.03 0.10 0.06 
Reading comprehension 

TOSREC -0.03 0.07 -0.15 0.20° 

SRI 0.32*** 0.34*** 0.13 0.49*** 


*p<.10.*p< .05. **p<.01.**p< .001. 
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Appendix G 
Student Characteristics 


Exhibit G1 


Characteristics of Students by Exit Status 


Characteristic 
Grade Level 
4 


5 
6 
7 
8 
Sex 
Male 
Female 
Free or Reduced-Price Meals 
None 
Reduced-price 
English Learner Status 
English only 
English learner 
Special Education Status 
None 
Specific learning disability 
Ethnicity 
Caucasian 
African American 
Hispanic 
School 
Coulter 
Heavenrich 
Herig 
Houghton 
Jerome 
Kempton 
Longfellow 


Loomis 


Merrill Park 
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Early 
(n= 11) 


Exit Status 


End of Year 
(n= 142) 


31 (100%) 
50 (93%) 
28 (90%) 
17 (89%) 
16 (89%) 


86 (92%) 
56 (93%) 


6 (86%) 
136 (93%) 


136 (93%) 
6 (100%) 


62 (89%) 
80 (96%) 


12 (75%) 


116 (95%) 
14 (93%) 
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Exhibit G1 (continued) 


Exit Status 
Early End of Year 
Characteristic (n= 11) (n= 142) 
Miller 0 (0%) 4 (100%) 
Rouse 1 (13%) 7 (88%) 
Stone 0 (0%) 4 (100%) 
Arthur Eddy 1 (9%) 10 (91%) 
Zilwaukee 0 (0%) 3 (100%) 
Ruben Daniels 0 (0%) 35 (100%) 
Thompson 5 (24%) 16 (76%) 


Note. Only 2 students were pre-decoders and are not included in this table. 
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Exhibit G2 
Characteristics of Students by Total Number of Topics Completed 


Number of Completed Topics 
Less Than 40 40 to 99 100 to 160 


Characteristic (n= 42) (n= 47) (n= 46) 
Grade Level 
4 20 (65%) 9 (29%) 2 (6%) 
5 16 (30%) 22 (41%) 16 (30%) 
6 10 (30%) 10 (30%) 13 (39%) 
7 2 (11%) 9 (47%) 8 (42%) 
8 4 (22%) 7 (39%) 7 (39%) 
Sex 
Male 32 (34%) 34 (36%) 28 (30%) 
Female 20 (33%) 23 (38%) 18 (30%) 
Free or Reduced-Price Meals 
None 2 (29%) 3 (43%) 2 (29%) 
Reduced-price 50 (34%) 54 (386%) 44 (30%) 
English Learner Status 
English only 50 (34%) 55 (37%) 44 (30%) 
English learner 2 (33%) 2 (33%) 2 (33%) 
Special Education Status 
None 14 (20%) 24 (34%) 32 (46%) 
Specific learning disability 37 (44%) 33 (39%) 14 (17%) 
Ethnicity 
Caucasian 4 (25%) 5 (31%) 7 (44%) 
African American 43 (35%) 47 (38%) 34 (27%) 
Hispanic 5 (33%) 5 (33%) 5 (33%) 
School 
Coulter 2 (40%) 2 (40%) 1 (20%) 
Heavenrich 4 (100%) 0 (0%) 0 (0%) 
Herig 0 (0%) 4 (33%) 8 (67%) 
Houghton 2 (40%) 3 (60%) 0 (0%) 
Jerome 4 (57%) 3 (43%) 0 (0%) 
Kempton 0 (0%) 3 (60%) 2 (40%) 
Longfellow 3 (60%) 2 (40%) 0 (0%) 
Loomis 9 (50%) 6 (33%) 3 (17%) 
Merrill Park 3 (50%) 3 (50%) 0 (0%) 
exhibit continues 
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Exhibit G2 (continued) 
Number of Completed Topics 
Less Than 40 40 to 100 100 to 160 


Characteristic (n= 42) (n= 47) (n = 46) 
Miller 1 (25%) 3 (75%) 0 (0%) 
Rouse 2 (25%) 2 (25%) 4 (50%) 
Stone 2 (50%) 1 (25%) 1 (25%) 
Arthur Eddy 5 (45%) 3 (27%) 3 (27%) 
Zilwaukee 1 (33%) 1 (33%) 1 (33%) 
Ruben Daniels 11 (30%) 14 (38%) 12 (32%) 
Thompson 3 (14%) 7 (33%) 11 (52%) 
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Exhibit G3 
Characteristics of Students by Initial Decoding Status 


Decoding Status 


Beginning Developing 

Characteristic (n=71) (n= 82) 
Grade Level 

4 24 (77%) 7 (23%) 

5 26 (49%) 27 (51%) 

6 11 (33%) 22 (67%) 

7 4 (22%) 14 (78%) 

8 6 (33%) 12 (67%) 
Sex 

Male 45 (49%) 47 (51%) 

Female 26 (43%) 35 (57%) 
Free or Reduced-Price Meals 

None 4 (57%) 3 (43%) 

Reduced-price 67 (46%) 79 (54%) 
English Learner Status 

English only 68 (46%) 79 (54%) 

English learner 3 (50%) 3 (50%) 
Special Education Status 

None 27 (38%) 44 (62%) 

Specific learning disability 44 (54%) 38 (46%) 
Ethnicity 

Caucasian 9 (56%) 7 (44%) 

African American 53 (43%) 69 (57%) 

Hispanic 9 (60%) 6 (40%) 
School 

Coulter 3 (60%) 2 (40%) 

Heavenrich 2 (50%) 2 (50%) 

Herig 6 (50%) 6 (50%) 

Houghton 5 (100%) 0 (0%) 

Jerome 5 (71%) 2 (29%) 

Kempton 4 (80%) 1 (20%) 

Longfellow 1 (25%) 3 (75%) 

Loomis 9 (50%) 9 (50%) 

Merrill Park 4 (67%) 2 (33%) 

exhibit continues 
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Exhibit G3 (continued) 
Decoding Status 


Beginning Developing 
Characteristic (n= 71) (n= 82) 
Miller 2 (50%) 2 (50%) 
Rouse 5 (63%) 3 (38%) 
Stone 3 (75%) 1 (25%) 
Arthur Eddy 4 (36%) 7 (64%) 
Zilwaukee 1 (383%) 2 (67%) 
Ruben Daniels 11 (381%) 25 (69%) 
Thompson 6 (29%) 15 (71%) 


Note. Only 2 students were pre-decoders and are not included in this table. 
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Appendix H 
Baseline Equivalence Tests 


Exhibit H1 
Baseline Equivalence of Exit Status Groups 
on Outcome Measures and Demographic Characteristics 


Exit Status 
Early End-of-Year 

Baseline Characteristic n M SD n M SD Pp 
TOSREC? 11. 27.40 13.82 141 13.68 13.04 .001 
CTOPP Elision® 11.21.27 5.59 142 17.83 5.83  .061 
TOWRE Sight Word Efficiency” 11 63.68 5.06 144 50.18 15.33 .004 
TOWRE Phonetic Decoding 11 30.68 7.23 142 19.63 10.99 .001 
Efficiency 

SRI° 11. 594.27 242.77 142 277.52 259.45 .000 
SPI Letter Name Accuracy*® 11 10.73 0.65 142 10.82 0.45 .508 
SPI Sight Word Accuracy*® 11 23.45 2.34 142 18.68 4.96 .002 
SPI Sight Word Fluency*® 11 9.00 3.82 142 5.98 4.09 .019 
SPI Nonsense Word Accuracy*® 11 21.09 3.99 142 15.82 4.24  .000 
SPI Nonsense Word Fluency*® 11 7.27 2.69 142 4.96 3.32 .027 
SPI Overall Accuracy® 11 44.55 4.57 142 34.51 7.84 — .000 
SPI Overall Fluency® 11 16.27 5.92 142 10.94 6.37 .008 
Female 11 .36 50 142 39 49 842 
Black 11 55 52 142 82 39 ~§=©.031 
Hispanic 11 .09 30 142 10 30 = -.935 
SPED Status 11 .27 7 142 56 50 .063 
ELL Status 11 .00 .00 142 .04 .20 490 


*Pretest assessment was analyzed using normal curve equivalent scores that ranged from 1 to 99. 


’Assessment was analyzed using raw scores. “Assessment was analyzed using scaled scores. 
“Assessment was analyzed using Lexile scores. 
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Exhibit H2 
Baseline Equivalence of Topic Completion Groups 
on Outcome Measures and Demographic Characteristics 


Number of Completed Topics 


. Less Than 40 40 to 99 100 to 160 

Baseline 

Characteristic n M SD n M SD n M SD p 
TOSREC?* 51 8.88 10.95 57 13.99 11.94 46 22.58 14.51 .000 
CTOPP Elision® 52 16.85 5.58 57 17.07 5.41 46 20.76 5.84 .001 
TOWRE Sight Word 51 39.07 12.90 57 52.95 13.50 46 62.28 8.43 .000 
Efficiency 

TOWRE Phonetic 52 14.28 7.26 57 20.03 11.55 46 27.83 9.60 .000 
Decoding 

Efficiency” 

SRI? 52 150.88 207.64 57 297.19 246.39 46 484.28 255.68 .000 
SPI Letter Name 52 10.85 0.36 57 10.74 0.58 46 10.89 0.38 .213 
Accuracy® 

SPI Sight Word 52 16.13 5.10 57 19.32 4.50 46 22.17 3.24 .000 
Accuracy® 

SPI Sight Word 52 3.42 3.06 57 6.40 3.76 46 9.00 3.57 .000 
Fluency® 

SPI Nonsense Word 52 = 13.75 3.95 57 15.58 3.74 46 19.96 3.23 .000 
Accuracy® 

SPI Nonsense Word 52 3.15 2.49 57 5.04 3.05 46 7.43 3.08 .000 
Fluency® 

SPI Overall 52 29.88 7.63 57 34.89 6.76 46 42.13 4.57 .001 
Accuracy® 

SPI Overall 52 6.58 4.56 57 11.44 5.62 46 16.43 5.14 .000 
Fluency® 

Female 52 .38 49 57 40 49 46 39 .49 .980 
Black 5283 38 57 82 38 46 74 44 474 
Hispanic 52 10 30 57 .09 .29 46 1 31.939 
SPED Status 52 71 46 57 .58 .50 46 .30 .47 —.000 
ELL Status 52 .04 19 57 .04 19 46 .04 21 977 


*Pretest assessment was analyzed using normal curve equivalent scores that ranged from 1 to 99. "Assessment 
was analyzed using raw scores. “Assessment was analyzed using scaled scores. “Assessment was analyzed using 
Lexile scores. 
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Exhibit H3 
Baseline Equivalence of Initial Decoding Status Groups 
on Outcome Measures and Demographic Characteristics 


Decoding Status 


Beginning Decoder Developing Decoder 

Baseline Characteristic n M SD n M SD Pp 
TOSREC?® 70 =«12.11 13.62 82 17.23 13.12 .066 
CTOPP Elision® 71 17.21 5.71 82 19.00 5.82 .054 
TOWRE Sight Word Efficiency” 70 = 42.71 14.65 82 58.61 11.06  .000 
TOWRE Phonetic Decoding 71 15.91 8.26 82 24.66 11.48 .000 
Efficiency” 

SRI° 71 205.92 233.49 82 389.55 272.77 .000 
SPI Letter Name Accuracy® 71 10.79 0.44 82 10.84 0.48 .672 
SPI Sight Word Accuracy® 71 16.79 4.89 82 21.16 4.12 .000 
SPI Sight Word Fluency* 71 2.97 2.49 82 9.00 3.04 .000 
SPI Nonsense Word Accuracy 71 15.08 4.00 82 17.52 4.28  .000 
SPI Nonsense Word Fluency*® 71 2.77 2.33 82 7.23 2.60 .000 
SPI Overall Accuracy® 71 31.87 7.65 82 38.68 7.00 .000 
SPI Overall Fluency® 71 5.75 4.03 82 16.23 3.50 .000 
Female 71 37 AQ 82 A3 50 .391 
Black 71 75 44 82 84 37 = .269 
Hispanic 71 13 .34 82 .07 26 .485 
SPED Status 71 62 49 82 46 50 .066 
ELL Status 71 .04 .20 82 04 19 .945 


*Pretest assessment was analyzed using normal curve equivalent scores that ranged from 1 to 99. 
’Assessment was analyzed using raw scores. “Assessment was analyzed using scaled scores. 
“Assessment was analyzed using Lexile scores. 
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Appendix | 
System 44 Software Usage Descriptives 


Exhibit !1 
System 44 Software Usage Descriptives 


Number of Number of 
Total Topics Total Hours on Topics Median Weeks in Sessions Per 
Total Sessions Completed Software Fast-Tracked | Session Time System 44 Week 

Subgroup n M SD M SD M SD M SD M SD M SD M SD 
Grade 
4 31 77.32 16.93 48.65 37.76 22.10 11.56 | 10.77 21.48 | 15.42 5.41 | 30.40 1.98 2.54 0.51 
5 54 75.63 14.10 73.80 50.69 22.25 9.72 | 32.31 36.39 | 18.22 5.75 | 29.79 2.82 2.55 0.44 
6 33 70.94 23.93 88.88 56.29 17.13 7.14 | 51.18 45.78 13.76 4.42 | 27.48 4.14 2.54 0.69 
7 19 81.68 17.64 | 100.47 48.01 24.38 7.73 | 48.63 35.64 18.89 3.98 | 28.41 4.58 2.91 0.55 
8 18 71.72 18.18 86.72 48.67 20.58 9.21 | 43.89 39.53 | 16.00 4.23 | 27.80 4.74 2.60 0.53 
Decoding Status 
Beginning decoder 71 75.13 17.19 48.37 38.53 22.13 9.91 8.38 21.94 16.49 5.64 | 29.75 2.42 2.52 0.53 
Developing decoder 82 75.87 18.87 | 102.76 47.26 20.68 9.12 59.60 3454 16.65 5.16 | 28.33 4.38 2.68 0.54 
FRL Status 
None 7 81.43 13.72 80.71 44.78 26.44 11.69 | 31.14 40.81 | 18.14 5.08 | 28.67 2.16 2.84 0.41 
Free or 148 74.97 18.26 76.56 51.66 | 20.95 9.39 35.57 38.91 16.46 5.36 | 29.04 3.71 2.58 0.55 
Reduced-Price 
ELL Status 
English only 149 75.11 18.21 7645 51.01 21.05 944 35.36 38.76 1645 536 28.98 3.69 2.59 0.55 
English learner 6 79.00 15.97 84.17 61.65 24.77 12.12 | 35.50 45.47 | 18.67 5.05 | 30.14 2.30 2.65 0.59 
SPED Status 
None 71 75.83 17.44 94.66 52.05 21.76 9.52 | 48.76 41.71 | 16.93 5.03 | 28.52 3.63 2.66 0.49 
Learning disability 84 74.77 18.73 61.61 45.62 20.72 9.57 | 24.05 32.45 | 16.20 5.61 | 29.45 3.63 2.54 0.59 
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Exhibit 11 (continued) 


Number of Number of 

Total Topics Total Hours on Topics Median Weeks in Sessions Per 

Total Sessions Completed Software Fast-Tracked | Session Time System 44 Week 

Subgroup n M SD M SD M SD M SD M SD M SD M SD 
Ethnicity 

Caucasian 16 76.06 20.41 95.69 56.66 | 26.86 11.67 | 44.44 45.17 | 21.44 6.24 | 27.25 5.34 | 2.82 0.61 

African American 124 74.57 18.27 72.85 49.83 | 19.97 8.66 34.00 37.62 15.73 4.95 29.18 3.46 2.55 0.54 

Hispanic 15 80.07 13.92 88.80 54.15 | 25.31 11.28 | 37.00 43.42 | 18.00 4.69 | 29.59 2.52 | 2.72 0.45 


Elementary School 


Coulter 5 87.80 10.99 50.80 37.56 21.54 5.38 | 18.20 26.44) 19.80 4.60 33.26 0.59) 264 0.35 
Heavenrich 4 44.00 8.76 28.75 10.14 9.57 1.89 12.00 13.86 9.50 4.51 29.29 1.72 1.50 0.26 
Herig 12 87.08 14.17 | 133.08 37.20 38.41 7.63 | 57.00 35.99 | 23.50 3.63 28.89 3.13 3.01 0.36 
Houghton 5 65.60 5.55 38.00 10.49 28.55 5.71 1.40 3.13 | 26.80 4.02 28.80 2.02 2.28 0.15 
Jerome 7 72.00 13.29 40.57 25.41 16.76 4.86 9.86 17.48 14.00 2.89 33.90 0.29 2.12 0.40 
Kempton 5 86.00 9.75 | 97.80 39.51 | 35.08 5.61 26.60 40.98 | 19.60 7.89 28.46 2.17) 3.02 0.24 
Longfellow 5 69.60 12.70 31.60 14.77 9.69 4.47 | 1440 13.15 | 15.00 3.08 31.17 1.21 2.22 0.34 
Loomis 18 71.67 9.11 60.72 42.58 15.59 3.56 | 29.06 36.88 | 13.94 2.75 28.41 3.02 2.53 0.24 
Merrill Park 6 92.00 7.13 42.50 15.35 20.86 1.35 8.50 12.79 13.50 1.76 31.02 0.96 2.96 0.18 
Miller 4 59.25 2.50 53.75 12.28 18.01 0.75 | 15.50 14.29 20.75 0.50 25.75 0.69 2.30 0.06 
Rouse 8 85.00 10.24 99.00 60.63 27.09 5.60 47.50 51.46) 17.25 3.77 30.50 3.32) 2.79 0.13 
Stone 4 86.00 455 64.00 5456 27.18 2.67 23.75 46.84 18.00 1.63 30.11 0.21 286 0.16 
K-8 School 

Arthur Eddy 11 65.18 12.60 70.64 59.39 14.84 3.11 | 40.64 51.32 | 1464 4.76 27.78 3.96 2.37 0.46 
Zilwaukee 3 69.33 3.06 86.00 57.42 18.62 0.53 | 46.67 49.74 17.67 2.31 30.24 0.66 2.29 0.07 
Middle School 

Ruben Daniels 37 73.78 23.50 75.84 47.87 18.20 9.09 | 37.62 30.47 13.32 3.99 29.31 3.03 2.47 0.67 
Thompson 21 77.95 20.10 | 112.24 52.47 23.76 7.47 | 61.33 47.85 19.38 4.07 25.85 5.25 3.03 0.49 


*Pre-decoders (n = 2) were excluded from the analyses. 
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